2022-03-08T22:44:10.647 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T22:44:10.649 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727281 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727281/orig.config.yaml 2022-03-08T22:44:10.682 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T22:44:10.748 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727281 branch: wip-yuri-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} email: null first_in_suite: false job_id: '6727281' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri-testing-2022-03-07-0958 ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-yuri-testing-2022-03-07-0958 sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 targets: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSOpiMsB0UujZKnK6gfqibz4GiK4zcbhxVQsUvtUwf1dUYFNKRYIBB9f5gmbF7h/dezG7IYBhgTMAmR4ve0Dt708uak/wreV+jIe9j5TnA0a/bULPdQYm/SqMpR/v3DGCqY/ZleYA6T+6QD0LJo+ECp5E03E99kUuet0VOR6KFe6ZjMmeiISQTvCbltiLSae6PdfKpWef6kd6k6JdILlQ1kgqYMNrksHN5k/OEFysRDuAD5RVrZkf+Jca8/e53ZcpvIKh+rkXoCK7MATRZI55oE0SQOjTooXcuHPi6a6Ph6yOZM9+ttTCVy4mM1+A+6Zgr9N6OgxAHQxaIQR8bA+wtxmepTALdFUyjgksBxqpBqJd40QbYhhGDzfVDLx3wL28lyXt4Xh8y1ferhPuRxYniPsMVSaaYdpKT8vLWnCLugezDPMsz6pjQnahcxgqvPh9erkC9V/pPZYu1i2z/plD1qM0Vk86H1tPtuNOiqWWLkgHj5Hbb79UkPicKDthrOnU= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-08_22:35:06 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T22:44:10.749 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa; will attempt to use it 2022-03-08T22:44:10.750 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks 2022-03-08T22:44:10.750 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T22:44:10.752 INFO:teuthology.task.internal:Checking packages... 2022-03-08T22:44:10.774 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '94cb3430b74fb099b8927e3e6ad50dfd969173f4' 2022-03-08T22:44:10.774 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T22:44:10.775 INFO:teuthology.packaging:ref: None 2022-03-08T22:44:10.775 INFO:teuthology.packaging:tag: None 2022-03-08T22:44:10.775 INFO:teuthology.packaging:branch: wip-yuri-testing-2022-03-07-0958 2022-03-08T22:44:10.776 INFO:teuthology.packaging:sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:44:10.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri-testing-2022-03-07-0958 2022-03-08T22:44:10.927 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11002-g94cb3430-1focal 2022-03-08T22:44:10.927 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T22:44:10.935 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T22:44:10.935 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T22:44:10.965 INFO:teuthology.task.internal:Saving configuration 2022-03-08T22:44:10.978 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T22:44:11.010 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T22:44:11.034 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727281', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:38:29.863945', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMUyTrPMfz1NgyrxFrglkRMXf4qVwb2ZG6zo68mSu15p'} 2022-03-08T22:44:11.035 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T22:44:11.060 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T22:44:11.060 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T22:44:11.163 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbb923369d8>, signals=[15]) 2022-03-08T22:44:11.164 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T22:44:11.181 INFO:teuthology.task.internal:Opening connections... 2022-03-08T22:44:11.181 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2022-03-08T22:44:11.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:44:11.238 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T22:44:11.251 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2022-03-08T22:44:11.361 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2022-03-08T22:44:11.362 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2022-03-08T22:44:11.369 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Ubuntu" 2022-03-08T22:44:11.369 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:44:11.370 INFO:teuthology.orchestra.run.smithi186.stdout:ID=ubuntu 2022-03-08T22:44:11.370 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE=debian 2022-03-08T22:44:11.370 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:44:11.371 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="20.04" 2022-03-08T22:44:11.371 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:44:11.371 INFO:teuthology.orchestra.run.smithi186.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:44:11.371 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:44:11.371 INFO:teuthology.orchestra.run.smithi186.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:44:11.372 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_CODENAME=focal 2022-03-08T22:44:11.372 INFO:teuthology.orchestra.run.smithi186.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:44:11.373 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-03-08T22:44:11.400 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T22:44:11.414 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T22:44:11.444 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T22:44:11.445 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:44:11.452 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T22:44:11.554 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T22:44:11.554 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:44:11.563 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T22:44:11.633 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T22:44:11.682 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T22:44:11.683 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T22:44:11.683 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-08T22:44:11.683 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T22:44:11.684 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2022-03-08T22:44:11.691 INFO:teuthology.orchestra.run.smithi186.stdout:5.4.0-97-generic 2022-03-08T22:44:11.691 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.4.0-97-generic 2022-03-08T22:44:11.691 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2022-03-08T22:44:11.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2022-03-08T22:44:12.079 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:44:12.090 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:44:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:44:12.497 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:44:12.619 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:44:12.756 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:44:12.993 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:44:13.107 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:44:13.244 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:44:13.261 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:44:13.262 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:44:13.263 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:44:13.349 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:44:13.356 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:44:13.363 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:44:13.363 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:44:13.367 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:44:13.368 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:44:13.370 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:44:13.492 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:44:13.521 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:44:13.524 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:44:13.525 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:44:13.526 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:44:13.528 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:44:13.528 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:44:13.529 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:44:13.529 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:44:13.530 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:44:13.535 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:44:13.539 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:44:13.541 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:44:13.546 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:44:13.546 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:44:13.547 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:44:13.642 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:44:13.643 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:44:13.643 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:44:13.644 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:44:13.645 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:44:13.645 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:44:13.646 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:44:13.742 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:44:16.371 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 10.2 MB in 2s (4,258 kB/s) 2022-03-08T22:44:17.615 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:44:17.638 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:44:17.707 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:44:17.880 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:44:17.880 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:44:18.029 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2022-03-08T22:44:18.029 INFO:teuthology.orchestra.run.smithi186.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:44:18.030 INFO:teuthology.orchestra.run.smithi186.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:44:18.030 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:44:18.030 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2022-03-08T22:44:18.031 INFO:teuthology.orchestra.run.smithi186.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:44:18.031 INFO:teuthology.orchestra.run.smithi186.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:44:18.077 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-08T22:44:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:44:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:44:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:44:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: thermald upower usbmuxd 2022-03-08T22:44:18.283 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:44:18.283 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:44:18.284 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:44:18.284 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:44:18.552 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:44:18.585 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:44:18.586 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:44:18.686 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:44:18.696 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:44:19.558 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:44:19.854 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:44:21.865 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:44:21.866 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:44:21.866 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:44:21.872 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:44:21.874 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:44:22.304 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 80.1 MB in 4s (21.1 MB/s) 2022-03-08T22:44:22.529 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:44:23.968 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:44:23.973 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:44:24.112 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:44:24.490 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:44:24.503 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:44:24.574 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:44:24.926 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:44:24.939 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:44:24.976 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:44:25.321 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:44:25.338 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:44:25.371 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:44:25.732 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:44:25.745 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:44:25.783 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:44:26.127 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:44:26.140 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:44:26.177 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:28.469 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:44:28.483 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:44:28.577 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:29.072 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:44:29.087 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:44:29.147 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:35.228 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:44:35.244 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:44:35.286 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:44:35.630 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:44:35.643 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:44:35.671 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:44:36.050 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package thermald. 2022-03-08T22:44:36.064 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:44:36.108 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:44:36.667 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package upower. 2022-03-08T22:44:36.681 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:44:36.736 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:44:37.145 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:44:37.153 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:44:37.190 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:44:37.505 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:44:37.654 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:44:37.779 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:44:37.930 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:44:38.056 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:44:38.174 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:44:38.333 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:44:38.836 INFO:teuthology.orchestra.run.smithi186.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:44:38.885 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:44:39.065 INFO:teuthology.orchestra.run.smithi186.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:44:39.079 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:44:39.080 INFO:teuthology.orchestra.run.smithi186.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:44:39.585 INFO:teuthology.orchestra.run.smithi186.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:44:39.801 INFO:teuthology.orchestra.run.smithi186.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:44:39.847 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:44:40.192 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:44:40.193 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:44:40.660 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:41.918 INFO:teuthology.orchestra.run.smithi186.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:44:41.918 INFO:teuthology.orchestra.run.smithi186.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:44:42.078 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:43.447 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:45.056 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:44:45.191 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:44:49.121 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:44:50.089 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:44:50.190 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:50.380 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:44:50.380 INFO:teuthology.orchestra.run.smithi186.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:13.978 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:45:13.978 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:45:13.989 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:45:14.007 INFO:teuthology.orchestra.run.smithi186.stdout:Generating grub configuration file ... 2022-03-08T22:45:14.287 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:45:14.332 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:14.741 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:45:14.753 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:45:17.357 INFO:teuthology.orchestra.run.smithi186.stdout:done 2022-03-08T22:45:17.726 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:45:17.758 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:45:17.758 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2022-03-08T22:45:17.759 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2022-03-08T22:45:17.759 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2022-03-08T22:45:17.760 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 19 2022-03-08T22:45:17.760 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:45:17.761 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2022-03-08T22:45:17.761 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:45:17.762 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:45:17.762 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:45:17.763 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:45:17.763 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2022-03-08T22:45:17.763 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2022-03-08T22:45:17.764 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:45:17.765 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2022-03-08T22:45:17.766 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:45:17.766 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:45:17.767 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:45:17.767 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T22:45:17.767 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:45:17.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2022-03-08T22:45:17.868 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2022-03-08T22:45:18.140 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:45:18.140 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:45:18.214 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:45:18.305 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:45:19.674 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:45:19.698 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:45:19.764 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:45:19.932 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:45:19.933 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:45:20.100 INFO:teuthology.orchestra.run.smithi186.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:45:20.100 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:45:20.102 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:45:20.120 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:45:20.121 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2022-03-08T22:45:20.121 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2022-03-08T22:45:20.121 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2022-03-08T22:45:20.121 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 19 2022-03-08T22:45:20.122 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:45:20.122 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2022-03-08T22:45:20.122 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:45:20.122 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:45:20.123 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:45:20.123 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:45:20.123 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2022-03-08T22:45:20.123 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2022-03-08T22:45:20.124 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:45:20.124 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2022-03-08T22:45:20.124 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2022-03-08T22:45:20.129 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.carqm7GxkD 2022-03-08T22:45:20.129 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub/grub.cfg /tmp/tmp.carqm7GxkD 2022-03-08T22:45:20.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.carqm7GxkD 2022-03-08T22:45:20.220 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.carqm7GxkD is 8KB 2022-03-08T22:45:20.234 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.carqm7GxkD 2022-03-08T22:45:20.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:45:20.296 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:45:20.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:45:20.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:45:20.334 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:45:20.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo update-grub 2022-03-08T22:45:21.543 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:45:21.545 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:45:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2022-03-08T22:45:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:45:21.851 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:22.240 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:45:22.252 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:45:23.880 INFO:teuthology.orchestra.run.smithi186.stderr:done 2022-03-08T22:45:23.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2022-03-08T22:45:53.910 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T22:45:53.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-08T22:45:53.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:46:12.427 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2022-03-08T22:46:42.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-08T22:46:42.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:00.811 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2022-03-08T22:47:30.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-08T22:47:30.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:31.064 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-03-08T22:47:31.579 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-08T22:47:31.580 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-08T22:47:31.580 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2022-03-08T22:47:31.585 INFO:teuthology.orchestra.run.smithi186.stdout:5.13.0-35-generic 2022-03-08T22:47:31.585 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:47:31.585 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:47:31.586 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:47:32.586 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T22:47:32.587 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:47:32.651 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2022-03-08T22:47:32.684 DEBUG:teuthology.parallel:result is None 2022-03-08T22:47:32.685 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T22:47:32.693 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T22:47:32.694 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:47:32.701 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T22:47:32.764 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T22:47:32.810 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T22:47:32.810 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:47:32.839 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T22:47:32.881 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T22:47:32.881 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:47:32.903 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:47:32.911 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:47:32.912 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T22:47:32.922 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T22:47:32.923 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:47:32.940 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T22:47:33.026 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T22:47:33.026 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:47:33.034 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:47:33.085 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:47:33.091 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:47:33.091 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:47:33.121 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2022-03-08T22:47:33.162 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T22:47:33.173 INFO:teuthology.task.internal:Starting timer... 2022-03-08T22:47:33.174 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T22:47:33.221 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T22:47:33.274 DEBUG:teuthology.task.selinux:Excluding smithi186: OS 'ubuntu' does not support SELinux 2022-03-08T22:47:33.275 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T22:47:33.275 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T22:47:33.276 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T22:47:33.276 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T22:47:33.345 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T22:47:33.346 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T22:47:33.367 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T22:47:33.368 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T22:50:30.912 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2022-03-08T22:50:30.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-08T22:50:30.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:50:30.969 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-03-08T22:50:31.088 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T22:50:31.107 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T22:50:31.108 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:50:31.110 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:50:31.140 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:50:31.140 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Command line: ntpd -gq 2022-03-08T22:50:31.141 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: proto: precision = 0.093 usec (-23) 2022-03-08T22:50:31.142 INFO:teuthology.orchestra.run.smithi186.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:50:31.143 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:50:31.143 INFO:teuthology.orchestra.run.smithi186.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:50:31.144 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:50:31.145 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:50:31.145 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:50:31.145 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:50:31.145 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listen normally on 3 enp3s0f1 172.21.15.186:123 2022-03-08T22:50:31.146 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listen normally on 4 lo [::1]:123 2022-03-08T22:50:31.146 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df5%5]:123 2022-03-08T22:50:31.146 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:31 ntpd[9053]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:50:32.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:32 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:32.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:32 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:32.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:32 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:32.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:32 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:32.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:32 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:32.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:32 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:33.140 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:33 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:33.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:33 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:33.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:33 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:33.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:33 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:33.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:33 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:33.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:33 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:34.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:34 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:34.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:34 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:34.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:34 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:34.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:34 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:34.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:34 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:34.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:34 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:35.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:35.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:35.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:35.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:35.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:35.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:35.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:35.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:35.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:35.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:35.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:35.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:35 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:36.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:36.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:36.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:36.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:36.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:36.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:36.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:36.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:36.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:36.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:36.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:36.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:36 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:37.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:37.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:37.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:37.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:37.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:37.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:37.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:37.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:37.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:37.143 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:37.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:37.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:37 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:38.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:38.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:38.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:38.141 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:38.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:38.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:38.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:38.142 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:38.143 INFO:teuthology.orchestra.run.smithi186.stdout: 8 Mar 22:50:38 ntpd[9053]: ntpd: time slew -0.000185 s 2022-03-08T22:50:38.143 INFO:teuthology.orchestra.run.smithi186.stdout:ntpd: time slew -0.000185s 2022-03-08T22:50:38.144 INFO:teuthology.orchestra.run.smithi186.stderr: 8 Mar 22:50:38 ntpd[9053]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:50:38.204 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:50:38.204 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2022-03-08T22:50:38.204 INFO:teuthology.orchestra.run.smithi186.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:38.204 INFO:teuthology.orchestra.run.smithi186.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:38.205 INFO:teuthology.orchestra.run.smithi186.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:38.205 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:38.206 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T22:50:38.221 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T22:50:38.222 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-08T22:50:38.222 DEBUG:teuthology.orchestra.run.smithi186:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:50:38.249 INFO:teuthology.orchestra.run.smithi186.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:50:38.256 INFO:teuthology.orchestra.run.smithi186.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:50:38.262 INFO:teuthology.orchestra.run.smithi186.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:50:38.320 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:50:38.321 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:50:38.377 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:50:38.378 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:50:38.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl restart docker || true 2022-03-08T22:50:39.450 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T22:50:39.457 INFO:tasks.kubeadm:172.21.15.186 is in 172.21.0.0/20 at pos 4025 2022-03-08T22:50:39.470 INFO:tasks.kubeadm:Pod subnet: 10.253.200.0/21 2022-03-08T22:50:39.471 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T22:50:39.471 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:50:39.500 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:39.501 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:50:39.501 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:39.737 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:50:39.737 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:50:39.809 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:50:39.891 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:50:41.230 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:50:41.403 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:50:41.404 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:50:41.490 INFO:teuthology.orchestra.run.smithi186.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:50:41.503 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:41.503 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:50:41.503 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:41.559 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:50:41.722 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:50:41.722 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:50:41.885 INFO:teuthology.orchestra.run.smithi186.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:50:41.886 INFO:teuthology.orchestra.run.smithi186.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:50:41.886 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-08T22:50:41.886 INFO:teuthology.orchestra.run.smithi186.stdout: apt-transport-https 2022-03-08T22:50:42.073 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:50:42.073 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 4,680 B of archives. 2022-03-08T22:50:42.074 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:50:42.074 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:50:42.295 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:50:42.295 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:50:42.296 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:50:42.311 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:50:42.311 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:50:42.311 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:50:42.317 INFO:teuthology.orchestra.run.smithi186.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:50:42.368 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 4,680 B in 0s (26.3 kB/s) 2022-03-08T22:50:42.456 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:50:42.500 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:50:42.503 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:50:42.547 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:50:42.833 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:50:43.399 INFO:teuthology.orchestra.run.smithi186.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:50:43.413 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:43.414 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:50:43.414 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:43.671 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:50:43.672 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:50:43.741 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:50:43.823 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:50:44.198 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:50:44.296 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:50:44.444 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 63.3 kB in 1s (71.8 kB/s) 2022-03-08T22:50:45.294 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:50:45.462 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:50:45.463 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:50:45.555 INFO:teuthology.orchestra.run.smithi186.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:50:45.570 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:45.570 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:50:45.571 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:50:45.625 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:50:45.799 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:50:45.800 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:50:45.942 INFO:teuthology.orchestra.run.smithi186.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:50:45.942 INFO:teuthology.orchestra.run.smithi186.stdout:bridge-utils set to manually installed. 2022-03-08T22:50:45.943 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2022-03-08T22:50:45.943 INFO:teuthology.orchestra.run.smithi186.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:50:45.944 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2022-03-08T22:50:45.944 INFO:teuthology.orchestra.run.smithi186.stdout: nftables 2022-03-08T22:50:45.994 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-08T22:50:45.995 INFO:teuthology.orchestra.run.smithi186.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:50:46.189 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:50:46.189 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:50:46.189 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:50:46.190 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:50:46.243 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:50:46.355 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:50:46.503 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:50:46.628 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:50:47.345 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:50:47.877 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:50:48.102 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:50:48.532 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:50:48.533 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:50:48.533 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:50:48.540 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:50:48.540 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:50:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:50:48.546 INFO:teuthology.orchestra.run.smithi186.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:50:48.600 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 73.6 MB in 2s (31.9 MB/s) 2022-03-08T22:50:48.687 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:50:48.731 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:50:48.735 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:50:48.769 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:50:49.113 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:50:49.127 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:50:49.164 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:50:50.224 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:50:50.239 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:50:50.306 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:50:50.586 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:50:50.600 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:50:50.644 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:50:52.063 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package socat. 2022-03-08T22:50:52.079 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:50:52.121 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:50:52.522 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:50:52.535 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:50:52.572 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:50:54.983 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:50:54.996 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:50:55.041 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:50:56.219 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:50:56.231 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:50:56.268 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:50:57.401 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:50:57.507 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:50:57.633 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:50:58.498 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:50:58.616 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:50:58.767 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:50:58.893 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:50:59.105 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:50:59.611 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:51:00.038 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:51:00.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:51:14.897 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:51:17.783 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:51:19.905 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:51:23.537 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:51:25.021 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:51:30.353 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:51:32.545 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:51:32.549 INFO:tasks.kubeadm:Token: abcdef.ygs3234q33z5n9ty 2022-03-08T22:51:32.549 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): '172.21.15.186'} 2022-03-08T22:51:32.550 DEBUG:teuthology.orchestra.run.smithi186:> sudo kubeadm init --node-name smithi186 --token abcdef.ygs3234q33z5n9ty --pod-network-cidr 10.253.200.0/21 2022-03-08T22:51:32.745 INFO:teuthology.orchestra.run.smithi186.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T22:51:32.745 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] Running pre-flight checks 2022-03-08T22:51:33.154 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T22:51:33.155 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T22:51:33.155 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T22:51:33.465 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T22:51:33.753 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "ca" certificate and key 2022-03-08T22:51:33.881 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T22:51:33.881 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi186] and IPs [10.96.0.1 172.21.15.186] 2022-03-08T22:51:34.058 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T22:51:34.169 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T22:51:34.336 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T22:51:34.467 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T22:51:34.741 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T22:51:34.742 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi186] and IPs [172.21.15.186 127.0.0.1 ::1] 2022-03-08T22:51:34.867 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T22:51:34.867 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi186] and IPs [172.21.15.186 127.0.0.1 ::1] 2022-03-08T22:51:35.080 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T22:51:35.379 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T22:51:35.430 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "sa" key and public key 2022-03-08T22:51:35.430 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T22:51:35.539 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T22:51:35.584 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T22:51:35.796 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T22:51:36.001 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T22:51:36.009 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:51:36.011 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:51:36.011 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:51:36.221 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T22:51:36.221 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T22:51:36.225 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T22:51:36.226 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T22:51:36.227 INFO:teuthology.orchestra.run.smithi186.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T22:51:36.230 INFO:teuthology.orchestra.run.smithi186.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T22:51:53.234 INFO:teuthology.orchestra.run.smithi186.stdout:[apiclient] All control plane components are healthy after 17.003279 seconds 2022-03-08T22:51:53.234 INFO:teuthology.orchestra.run.smithi186.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T22:51:53.258 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T22:51:53.258 INFO:teuthology.orchestra.run.smithi186.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T22:51:53.876 INFO:teuthology.orchestra.run.smithi186.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T22:51:53.877 INFO:teuthology.orchestra.run.smithi186.stdout:[mark-control-plane] Marking the node smithi186 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T22:51:53.877 INFO:teuthology.orchestra.run.smithi186.stdout:[mark-control-plane] Marking the node smithi186 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T22:51:54.445 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] Using token: abcdef.ygs3234q33z5n9ty 2022-03-08T22:51:54.445 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T22:51:54.479 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T22:51:54.500 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T22:51:54.508 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T22:51:54.568 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T22:51:54.572 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T22:51:54.621 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T22:51:55.051 INFO:teuthology.orchestra.run.smithi186.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T22:51:55.209 INFO:teuthology.orchestra.run.smithi186.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T22:51:55.210 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.211 INFO:teuthology.orchestra.run.smithi186.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T22:51:55.211 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.211 INFO:teuthology.orchestra.run.smithi186.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T22:51:55.211 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.212 INFO:teuthology.orchestra.run.smithi186.stdout: mkdir -p $HOME/.kube 2022-03-08T22:51:55.212 INFO:teuthology.orchestra.run.smithi186.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T22:51:55.212 INFO:teuthology.orchestra.run.smithi186.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T22:51:55.212 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.213 INFO:teuthology.orchestra.run.smithi186.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T22:51:55.213 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.213 INFO:teuthology.orchestra.run.smithi186.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T22:51:55.213 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.214 INFO:teuthology.orchestra.run.smithi186.stdout:You should now deploy a pod network to the cluster. 2022-03-08T22:51:55.214 INFO:teuthology.orchestra.run.smithi186.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T22:51:55.214 INFO:teuthology.orchestra.run.smithi186.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T22:51:55.214 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.215 INFO:teuthology.orchestra.run.smithi186.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T22:51:55.215 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:51:55.215 INFO:teuthology.orchestra.run.smithi186.stdout:kubeadm join 172.21.15.186:6443 --token abcdef.ygs3234q33z5n9ty \ 2022-03-08T22:51:55.215 INFO:teuthology.orchestra.run.smithi186.stdout: --discovery-token-ca-cert-hash sha256:c36ef793e58c804240cedfbfd254a144ccd687f8c916e2865adb96bea4da3e3e 2022-03-08T22:51:55.217 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:51:55.217 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T22:51:55.226 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T22:51:55.227 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T22:51:55.292 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:51:55.292 DEBUG:teuthology.orchestra.run.smithi186:> dd of=.kube/config 2022-03-08T22:51:55.299 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:51:55.300 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/root/.kube/config 2022-03-08T22:51:55.311 DEBUG:teuthology.orchestra.run.smithi186:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-08T22:51:58.373 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-08T22:51:58.430 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-08T22:51:58.535 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-08T22:51:58.570 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-08T22:51:58.618 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-08T22:51:58.756 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-08T22:51:58.811 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-08T22:51:58.901 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-08T22:51:58.977 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-08T22:51:59.066 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-08T22:51:59.119 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-08T22:51:59.229 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-08T22:51:59.261 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-08T22:51:59.311 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-08T22:51:59.405 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-08T22:51:59.467 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-08T22:51:59.551 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-08T22:51:59.604 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-08T22:51:59.746 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-08T22:51:59.805 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-08T22:51:59.921 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-08T22:51:59.979 INFO:teuthology.orchestra.run.smithi186.stdout:namespace/tigera-operator created 2022-03-08T22:51:59.989 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:51:59.994 INFO:teuthology.orchestra.run.smithi186.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-08T22:52:00.002 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/tigera-operator created 2022-03-08T22:52:00.071 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T22:52:00.139 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T22:52:00.156 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps/tigera-operator created 2022-03-08T22:52:00.162 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f - 2022-03-08T22:52:03.183 INFO:teuthology.orchestra.run.smithi186.stdout:installation.operator.tigera.io/default created 2022-03-08T22:52:03.189 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T22:52:03.190 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:52:03.190 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:52:03.195 INFO:tasks.kubeadm:Scratch devices on smithi186: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:52:03.196 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T22:52:03.222 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-08T22:52:03.223 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-08T22:52:03.223 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155869 s, 673 MB/s 2022-03-08T22:52:03.224 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T22:52:03.298 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-08T22:52:03.298 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-08T22:52:03.299 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163454 s, 642 MB/s 2022-03-08T22:52:03.300 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T22:52:03.326 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-08T22:52:03.327 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-08T22:52:03.327 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0152621 s, 687 MB/s 2022-03-08T22:52:03.328 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T22:52:03.355 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-08T22:52:03.355 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-08T22:52:03.356 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149907 s, 699 MB/s 2022-03-08T22:52:03.363 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T22:52:03.363 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T22:52:03.363 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f - 2022-03-08T22:52:04.268 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-1 created 2022-03-08T22:52:04.273 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-2 created 2022-03-08T22:52:04.309 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-3 created 2022-03-08T22:52:04.312 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-4 created 2022-03-08T22:52:04.323 INFO:teuthology.orchestra.run.smithi186.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T22:52:04.328 DEBUG:teuthology.orchestra.run.smithi186:> kubectl taint node smithi186 node-role.kubernetes.io/master- || true 2022-03-08T22:52:04.396 INFO:teuthology.orchestra.run.smithi186.stdout:node/smithi186 untainted 2022-03-08T22:52:04.398 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T22:52:04.399 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T22:52:04.411 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T22:52:04.411 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:52:04.412 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:52:04.420 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:52:04.420 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T22:52:04.467 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T22:52:04.467 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:04.467 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-08T22:52:04.468 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:04.468 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-08 22:52:03.229942003 +0000 2022-03-08T22:52:04.468 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-08 22:52:03.229942003 +0000 2022-03-08T22:52:04.468 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-08 22:52:03.229942003 +0000 2022-03-08T22:52:04.469 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-08T22:52:04.477 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-08T22:52:04.477 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-08T22:52:04.477 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000608798 s, 841 kB/s 2022-03-08T22:52:04.482 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T22:52:04.488 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T22:52:04.489 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:04.489 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-08T22:52:04.489 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:04.489 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-08 22:52:03.305942006 +0000 2022-03-08T22:52:04.490 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-08 22:52:03.305942006 +0000 2022-03-08T22:52:04.490 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-08 22:52:03.305942006 +0000 2022-03-08T22:52:04.490 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-08T22:52:04.496 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-08T22:52:04.496 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-08T22:52:04.496 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000483991 s, 1.1 MB/s 2022-03-08T22:52:04.500 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T22:52:04.506 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T22:52:04.506 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:04.506 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-08T22:52:04.506 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:04.507 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-08 22:52:03.333942006 +0000 2022-03-08T22:52:04.507 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-08 22:52:03.333942006 +0000 2022-03-08T22:52:04.507 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-08 22:52:03.333942006 +0000 2022-03-08T22:52:04.507 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-08T22:52:04.514 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-08T22:52:04.515 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-08T22:52:04.515 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000481181 s, 1.1 MB/s 2022-03-08T22:52:04.519 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T22:52:04.567 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T22:52:04.568 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:04.568 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-08T22:52:04.568 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:04.568 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-08 22:52:03.361942007 +0000 2022-03-08T22:52:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-08 22:52:03.361942007 +0000 2022-03-08T22:52:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-08 22:52:03.361942007 +0000 2022-03-08T22:52:04.569 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-08T22:52:04.577 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-08T22:52:04.578 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-08T22:52:04.578 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000699105 s, 732 kB/s 2022-03-08T22:52:04.582 DEBUG:teuthology.orchestra.run.smithi186:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T22:52:04.684 INFO:teuthology.orchestra.run.smithi186.stdout:loop 2022-03-08T22:52:04.685 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_1... 2022-03-08T22:52:04.686 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T22:52:04.703 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-08T22:52:04.729 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_11 2022-03-08T22:52:04.748 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_2... 2022-03-08T22:52:04.749 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T22:52:04.770 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-08T22:52:04.791 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_21 2022-03-08T22:52:04.815 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_3... 2022-03-08T22:52:04.816 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T22:52:04.879 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-08T22:52:04.906 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_31 2022-03-08T22:52:04.928 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_4... 2022-03-08T22:52:04.929 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T22:52:04.948 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-08T22:52:04.973 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_41 2022-03-08T22:52:04.998 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:52:04.999 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:52:05.006 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme list 2022-03-08T22:52:05.059 INFO:teuthology.orchestra.run.smithi186.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:52:05.059 INFO:teuthology.orchestra.run.smithi186.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:52:05.060 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme0n1 CVFT623300CB400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T22:52:05.060 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme1n1 b1a5a33c08051c89 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:05.060 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme2n1 2e085d41fc4e3f25 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:05.060 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme3n1 a3bcb430017fbda8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:05.060 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme4n1 f11e6f4f99590806 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:05.061 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T22:52:05.062 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:52:05.062 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/scratch_devs 2022-03-08T22:52:05.075 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T22:52:05.142 INFO:tasks.rook:Rook start 2022-03-08T22:52:05.143 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-08T22:52:05.143 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:52:05.144 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-08T22:52:05.144 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-08T22:52:05.152 INFO:teuthology.orchestra.run.smithi186.stderr:Cloning into 'rook'... 2022-03-08T22:52:07.327 INFO:teuthology.orchestra.run.smithi186.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-08T22:52:07.328 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.328 INFO:teuthology.orchestra.run.smithi186.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-08T22:52:07.328 INFO:teuthology.orchestra.run.smithi186.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-08T22:52:07.328 INFO:teuthology.orchestra.run.smithi186.stderr:state without impacting any branches by switching back to a branch. 2022-03-08T22:52:07.329 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.329 INFO:teuthology.orchestra.run.smithi186.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-08T22:52:07.329 INFO:teuthology.orchestra.run.smithi186.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-08T22:52:07.329 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.329 INFO:teuthology.orchestra.run.smithi186.stderr: git switch -c 2022-03-08T22:52:07.330 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.330 INFO:teuthology.orchestra.run.smithi186.stderr:Or undo this operation with: 2022-03-08T22:52:07.330 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.330 INFO:teuthology.orchestra.run.smithi186.stderr: git switch - 2022-03-08T22:52:07.331 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.331 INFO:teuthology.orchestra.run.smithi186.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-08T22:52:07.331 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-08T22:52:07.424 INFO:tasks.rook:/home/teuthworker 2022-03-08T22:52:07.425 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T22:52:07.430 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T22:52:07.430 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T22:52:07.431 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T22:52:07.431 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:07.431 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2022-03-08T22:52:07.431 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:52:07.432 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:07.432 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2022-03-08T22:52:07.432 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:52:07.432 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:07.432 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2022-03-08T22:52:07.433 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:52:07.433 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:07.433 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T22:52:07.433 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:07.433 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-08T22:52:07.434 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42906sec preferred_lft 42906sec 2022-03-08T22:52:07.434 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::ec4:7aff:fe88:6df5/64 scope link 2022-03-08T22:52:07.434 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:07.434 INFO:teuthology.orchestra.run.smithi186.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T22:52:07.435 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 02:42:15:cb:f0:b7 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:07.435 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T22:52:07.435 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:07.436 DEBUG:teuthology.orchestra.run.smithi186:> which systemctl 2022-03-08T22:52:07.477 INFO:teuthology.orchestra.run.smithi186.stdout:/usr/bin/systemctl 2022-03-08T22:52:07.477 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T22:52:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T22:52:07.481 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-03-08T22:52:07.510 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-08T22:52:07.549 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-08T22:52:07.553 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:52:07.553 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T22:52:07.608 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:52:07.621 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:52:07.621 DEBUG:teuthology.orchestra.run.smithi186:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-08T22:52:07.628 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-08T22:52:07.668 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:52:07.668 DEBUG:teuthology.orchestra.run.smithi186:> dd of=operator.yaml 2022-03-08T22:52:07.682 INFO:tasks.rook:Deploying operator 2022-03-08T22:52:07.697 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:52:07.764 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:52:07.774 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-08T22:52:08.075 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T22:52:08.098 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T22:52:08.456 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T22:52:08.489 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T22:52:08.768 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T22:52:08.858 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T22:52:09.138 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T22:52:09.178 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T22:52:09.394 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T22:52:09.399 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T22:52:09.465 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T22:52:09.588 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T22:52:09.608 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T22:52:09.656 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T22:52:09.787 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-08T22:52:09.814 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-08T22:52:09.878 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-08T22:52:09.990 INFO:teuthology.orchestra.run.smithi186.stdout:namespace/rook-ceph created 2022-03-08T22:52:10.032 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:52:10.124 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-08T22:52:10.165 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-08T22:52:10.211 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-08T22:52:10.359 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:52:10.362 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:10.391 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:10.402 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:52:10.434 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:52:10.483 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:52:10.486 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-system created 2022-03-08T22:52:10.502 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:10.514 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:10.567 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:52:10.665 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T22:52:10.669 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T22:52:10.689 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T22:52:10.737 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:11.083 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:11.085 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:52:11.120 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:52:11.288 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:52:11.322 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:52:11.600 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:11.604 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:52:11.633 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:52:11.656 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:52:11.663 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:11.699 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:52:11.783 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:52:11.784 INFO:teuthology.orchestra.run.smithi186.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T22:52:11.787 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T22:52:11.817 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T22:52:11.829 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T22:52:11.858 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T22:52:11.984 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T22:52:11.987 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T22:52:12.002 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T22:52:12.025 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T22:52:12.035 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T22:52:12.076 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T22:52:12.133 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:52:12.139 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T22:52:12.169 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T22:52:12.210 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T22:52:12.227 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:52:12.299 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T22:52:12.320 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T22:52:12.330 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T22:52:12.341 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T22:52:12.362 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T22:52:12.488 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:52:12.494 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T22:52:12.521 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T22:52:12.554 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T22:52:12.579 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:52:12.641 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T22:52:12.647 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:52:12.655 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:52:12.663 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T22:52:12.688 INFO:teuthology.orchestra.run.smithi186.stdout:configmap/rook-ceph-operator-config created 2022-03-08T22:52:12.698 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T22:52:12.702 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:12.831 INFO:teuthology.orchestra.run.smithi186.stderr:No resources found in rook-ceph namespace. 2022-03-08T22:52:22.836 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:22.900 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:22.900 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 Pending 0 10s 2022-03-08T22:52:32.903 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:33.060 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:33.060 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 Pending 0 21s 2022-03-08T22:52:43.063 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:43.473 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:43.474 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 Pending 0 31s 2022-03-08T22:52:53.476 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:53.969 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:53.970 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 Pending 0 41s 2022-03-08T22:53:03.972 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:04.033 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:04.033 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 ContainerCreating 0 52s 2022-03-08T22:53:14.036 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:14.108 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:14.108 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 ContainerCreating 0 62s 2022-03-08T22:53:24.111 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:24.260 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:24.260 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 ContainerCreating 0 72s 2022-03-08T22:53:34.264 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:34.418 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:34.418 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 0/1 ContainerCreating 0 82s 2022-03-08T22:53:44.422 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:44.487 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:44.487 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-tjmtw 1/1 Running 0 92s 2022-03-08T22:53:44.489 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-tjmtw 2022-03-08T22:53:44.496 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:53:44.497 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:53:44.504 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-08T22:53:44.504 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T22:53:44.504 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T22:53:44.505 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T22:53:44.505 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T22:53:44.506 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T22:53:44.506 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T22:53:44.506 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T22:53:44.507 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T22:53:44.509 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f - 2022-03-08T22:53:44.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.603489 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T22:53:44.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.603600 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-08T22:53:44.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.603604 I | cephcmd: starting Rook-Ceph operator 2022-03-08T22:53:44.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.891475 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-08T22:53:44.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.948061 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T22:53:44.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.982267 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T22:53:44.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.985083 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-08T22:53:44.579 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.987164 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T22:53:44.579 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.989802 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T22:53:44.579 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.989818 I | operator: watching all namespaces for ceph cluster CRs 2022-03-08T22:53:44.579 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.989953 I | operator: setting up the controller-runtime manager 2022-03-08T22:53:44.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.992805 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-08T22:53:44.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:38.996549 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T22:53:44.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397249 I | ceph-cluster-controller: successfully started 2022-03-08T22:53:44.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397339 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T22:53:44.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397364 I | ceph-crashcollector-controller: successfully started 2022-03-08T22:53:44.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397493 I | ceph-block-pool-controller: successfully started 2022-03-08T22:53:44.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397606 I | ceph-object-store-user-controller: successfully started 2022-03-08T22:53:44.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397719 I | ceph-object-realm-controller: successfully started 2022-03-08T22:53:44.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397830 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T22:53:44.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.397968 I | ceph-object-zone-controller: successfully started 2022-03-08T22:53:44.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.398171 I | ceph-object-controller: successfully started 2022-03-08T22:53:44.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.398314 I | ceph-file-controller: successfully started 2022-03-08T22:53:44.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.398415 I | ceph-nfs-controller: successfully started 2022-03-08T22:53:44.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.398552 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T22:53:44.583 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.398656 I | ceph-client-controller: successfully started 2022-03-08T22:53:44.583 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.398752 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T22:53:44.583 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:39.400242 I | operator: starting the controller-runtime manager 2022-03-08T22:53:45.495 INFO:teuthology.orchestra.run.smithi186.stdout:configmap/rook-config-override created 2022-03-08T22:53:45.499 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-03-08T22:53:45.499 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:53:45.500 DEBUG:teuthology.orchestra.run.smithi186:> dd of=cluster.yaml 2022-03-08T22:53:45.505 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f cluster.yaml 2022-03-08T22:53:48.065 INFO:teuthology.orchestra.run.smithi186.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T22:53:48.069 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.071953 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-08T22:53:48.069 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.072133 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T22:53:48.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.081293 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:53:48.082 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.081998 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T22:53:48.083 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-08T22:53:48.083 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-08T22:53:48.083 INFO:tasks.rook.operator.smithi186.stdout: Placement: nil, 2022-03-08T22:53:48.083 INFO:tasks.rook.operator.smithi186.stdout: Network: v1.NetworkSpec{ 2022-03-08T22:53:48.084 INFO:tasks.rook.operator.smithi186.stdout: Provider: "", 2022-03-08T22:53:48.084 INFO:tasks.rook.operator.smithi186.stdout: Selectors: nil, 2022-03-08T22:53:48.084 INFO:tasks.rook.operator.smithi186.stdout: HostNetwork: false, 2022-03-08T22:53:48.084 INFO:tasks.rook.operator.smithi186.stdout:- IPFamily: "", 2022-03-08T22:53:48.084 INFO:tasks.rook.operator.smithi186.stdout:+ IPFamily: "IPv4", 2022-03-08T22:53:48.085 INFO:tasks.rook.operator.smithi186.stdout: DualStack: false, 2022-03-08T22:53:48.085 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-08T22:53:48.085 INFO:tasks.rook.operator.smithi186.stdout: Resources: nil, 2022-03-08T22:53:48.085 INFO:tasks.rook.operator.smithi186.stdout: PriorityClassNames: nil, 2022-03-08T22:53:48.086 INFO:tasks.rook.operator.smithi186.stdout: ... // 16 identical fields 2022-03-08T22:53:48.086 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-08T22:53:48.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.092667 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T22:53:48.095 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:53:48.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.106571 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T22:53:48.106 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T22:53:48.161 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.164335 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-08T22:53:48.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.164441 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T22:53:48.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.167678 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T22:53:48.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.175544 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T22:53:48.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.177466 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T22:53:48.178 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.180867 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:53:48.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.184063 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T22:53:48.360 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.362575 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-08T22:53:48.398 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T22:53:48.401 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T22:53:48.402 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:53:48.553 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:48.554 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 Pending 0 0s 2022-03-08T22:53:48.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:48.685907 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-08T22:53:49.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:49.260490 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-08T22:53:49.262 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:49.264330 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-08T22:53:49.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:49.283836 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-08T22:53:49.530 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:49.533423 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T22:53:49.723 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:49.725241 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-08T22:53:49.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:49.884786 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T22:53:50.084 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:50.087120 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T22:53:50.084 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:50.087143 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-08T22:53:50.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:53:50.284715 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:53:53.555 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:53:53.608 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:53.608 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 5s 2022-03-08T22:53:58.611 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:53:58.684 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:58.684 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 10s 2022-03-08T22:54:03.687 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:03.752 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:03.752 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 15s 2022-03-08T22:54:08.755 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:08.817 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:08.817 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 20s 2022-03-08T22:54:13.820 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:14.123 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:14.123 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 26s 2022-03-08T22:54:19.126 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:19.247 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:19.248 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 31s 2022-03-08T22:54:24.252 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:24.502 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:24.503 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 36s 2022-03-08T22:54:27.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.419638 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-08T22:54:27.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.442415 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T22:54:27.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.445778 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T22:54:27.490 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.485900 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T22:54:27.838 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.837855 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T22:54:27.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.926682 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:54:27.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.926710 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:54:27.929 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:27.926720 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:54:28.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.025763 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:54:28.027 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.025868 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T22:54:28.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.028289 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:54:28.068 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.067898 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:54:28.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.109511 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T22:54:28.113 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.112056 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 22:54:28.068632005 +0000 UTC m=+49.522909535 LastTransitionTime:2022-03-08 22:54:28.068631658 +0000 UTC m=+49.522909266}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:54:28.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.118301 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T22:54:28.188 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.187230 I | op-mon: start running mons 2022-03-08T22:54:28.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.243382 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T22:54:28.555 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.555171 I | op-mon: creating mon secrets for a new cluster 2022-03-08T22:54:28.642 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:28.641453 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T22:54:29.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:29.200080 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:54:29.506 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:29.533 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:29.532070 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:54:29.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:29.532134 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T22:54:29.586 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:29.586 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 41s 2022-03-08T22:54:29.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:29.620525 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T22:54:29.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:29.821208 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-08T22:54:30.023 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:30.022669 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T22:54:30.221 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:30.220927 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T22:54:31.023 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:31.022219 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:31.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:31.421682 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:31.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:31.620301 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:54:31.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:31.620597 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:54:31.880 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:31.879790 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:54:32.221 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:32.220603 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:54:32.622 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:32.621486 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:54:33.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:33.021387 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:54:33.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:33.422382 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T22:54:33.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:33.821688 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:34.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:34.021132 I | op-mon: targeting the mon count 1 2022-03-08T22:54:34.327 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:34.325528 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:34.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:34.336817 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T22:54:34.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:34.336900 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:54:34.481 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:34.480382 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:54:34.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:34.504031 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T22:54:34.588 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:34.896 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:34.897 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 46s 2022-03-08T22:54:35.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:35.093621 I | op-mon: parsing mon endpoints: 2022-03-08T22:54:35.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:35.093674 W | op-mon: ignoring invalid monitor 2022-03-08T22:54:35.122 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:35.120949 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:54:35.305 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:35.305115 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:54:35.863 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:35.861103 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:54:36.408 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:36.408145 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T22:54:36.712 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:36.711715 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:36.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:36.981897 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:37.389 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:37.388446 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:54:37.666 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:37.599973 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T22:54:38.254 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:38.253345 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T22:54:38.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:38.503797 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T22:54:38.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:38.596455 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-08T22:54:39.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:39.039442 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T22:54:39.903 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:40.439 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:40.439 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 52s 2022-03-08T22:54:45.441 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:45.599 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:45.599 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 57s 2022-03-08T22:54:49.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.229372 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:54:49.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.229394 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:54:49.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.229405 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:54:49.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.290844 I | op-mon: parsing mon endpoints: 2022-03-08T22:54:49.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.290874 W | op-mon: ignoring invalid monitor 2022-03-08T22:54:49.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.497441 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:54:49.502 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.497808 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:54:49.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.620337 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T22:54:49.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T22:54:49.613+0000 7fd05c913700 -1 failed for service _ceph-mon._tcp 2022-03-08T22:54:49.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T22:54:49.613+0000 7fd05c913700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T22:54:49.621 INFO:tasks.rook.operator.smithi186.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-03-08T22:54:49.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:49.620358 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:54:50.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.169722 I | op-mon: start running mons 2022-03-08T22:54:50.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.310656 I | op-mon: parsing mon endpoints: 2022-03-08T22:54:50.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.310681 W | op-mon: ignoring invalid monitor 2022-03-08T22:54:50.393 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.392312 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T22:54:50.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.482117 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:54:50.488 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.487249 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:54:50.602 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:50.658 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:50.659 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 62s 2022-03-08T22:54:50.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:50.973591 I | op-mon: targeting the mon count 1 2022-03-08T22:54:51.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:51.085449 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T22:54:55.661 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:55.762 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 67s 2022-03-08T22:54:56.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:56.575697 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi186 2022-03-08T22:54:56.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:56.575729 I | op-mon: assignmon: mon a assigned to node smithi186 2022-03-08T22:54:56.579 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:56.579059 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T22:54:56.592 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:56.583663 I | op-mon: creating mon a 2022-03-08T22:54:56.754 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:56.753873 I | op-mon: mon "a" endpoint is [v2:10.107.60.203:3300,v1:10.107.60.203:6789] 2022-03-08T22:54:57.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.281050 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:-1] 2022-03-08T22:54:57.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.281198 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T22:54:57.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.281384 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T22:54:57.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.435170 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:54:57.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.435500 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:54:57.458 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.458207 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T22:54:57.632 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.629535 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T22:54:57.819 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.818850 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T22:54:57.820 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:57.819081 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:54:58.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:58.033134 I | op-mon: mon a is not yet running 2022-03-08T22:54:58.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:54:58.033157 I | op-mon: mons running: [] 2022-03-08T22:55:00.766 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:00.899 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:00.899 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 0/1 ContainerCreating 0 72s 2022-03-08T22:55:05.902 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:06.063 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:06.063 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-9xrgh 1/1 Running 0 78s 2022-03-08T22:55:06.064 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T22:55:06.065 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status -f json 2022-03-08T22:55:06.299 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T22:55:06.299 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-08T22:55:06.297+0000 7fd7237d1700 -1 failed for service _ceph-mon._tcp 2022-03-08T22:55:06.299 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-08T22:55:06.297+0000 7fd7237d1700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T22:55:06.300 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T22:55:06.302 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-08T22:55:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T22:55:16.305 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status -f json 2022-03-08T22:55:17.567 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:55:17.568 INFO:teuthology.orchestra.run.smithi186.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:17.568 INFO:teuthology.orchestra.run.smithi186.stderr:pg stat 2022-03-08T22:55:17.569 INFO:teuthology.orchestra.run.smithi186.stderr:pg getmap 2022-03-08T22:55:17.569 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:17.569 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:17.569 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_pools_json 2022-03-08T22:55:17.570 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-pool [...] 2022-03-08T22:55:17.570 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:17.570 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:17.570 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls [] [...] 2022-03-08T22:55:17.571 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:17.571 INFO:teuthology.orchestra.run.smithi186.stderr:Error EINVAL: invalid command 2022-03-08T22:55:17.571 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 22 2022-03-08T22:55:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:18.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.167077 I | op-mon: mons running: [a] 2022-03-08T22:55:18.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.427983 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:18.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.428008 I | op-mon: mons created: 1 2022-03-08T22:55:18.654 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.654837 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:55:18.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.663054 I | op-mon: mons running: [a] 2022-03-08T22:55:18.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.876415 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:18.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:18.876450 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:19.149 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:19.149685 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:19.150 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:19.149715 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:19.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:19.557444 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:19.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:19.557482 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:19.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:19.845231 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:19.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:19.845267 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:20.135 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.135829 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:20.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.135861 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:55:20.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.404362 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:55:20.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.404393 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:55:20.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.645590 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:55:20.646 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.645626 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:55:20.921 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:20.921033 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:55:21.216 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:21.216236 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:55:21.514 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:21.514266 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:55:21.929 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:21.929577 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:55:21.929 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:21.929601 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:55:22.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.269080 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:55:22.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.497619 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:55:22.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.497654 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:55:22.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.497661 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:55:22.798 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.798657 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:55:22.811 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.811204 I | op-mgr: start running mgr 2022-03-08T22:55:22.811 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:22.811245 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:55:27.575 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status -f json 2022-03-08T22:55:28.034 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:55:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:28.084 INFO:teuthology.orchestra.run.smithi186.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:28.085 INFO:teuthology.orchestra.run.smithi186.stderr:pg stat 2022-03-08T22:55:28.085 INFO:teuthology.orchestra.run.smithi186.stderr:pg getmap 2022-03-08T22:55:28.085 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:28.085 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:28.086 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_pools_json 2022-03-08T22:55:28.086 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-pool [...] 2022-03-08T22:55:28.086 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:28.086 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:28.087 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls [] [...] 2022-03-08T22:55:28.087 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:28.087 INFO:teuthology.orchestra.run.smithi186.stderr:Error EINVAL: invalid command 2022-03-08T22:55:28.087 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 22 2022-03-08T22:55:32.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:32.696060 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T22:55:32.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:32.698647 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:55:32.999 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.000133 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:55:33.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.000157 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T22:55:33.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.000166 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:55:33.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.046351 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:55:33.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.080243 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T22:55:33.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.080273 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:55:33.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.080278 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:55:33.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.522189 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-08T22:55:33.531 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.531941 I | op-mgr: successful modules: dashboard 2022-03-08T22:55:33.591 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.591728 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T22:55:33.591 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.591754 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:55:33.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.597277 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:55:33.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.597299 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:55:33.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.597318 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:55:33.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.634749 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T22:55:33.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.634768 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:55:33.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.635849 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-08T22:55:33.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.635974 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:55:33.637 INFO:tasks.rook.operator.smithi186.stdout:I0308 22:55:33.635961 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:55:33.642 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.641484 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:55:33.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.705157 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:55:33.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:33.766938 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:34.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:34.945933 I | op-mgr: successful modules: prometheus 2022-03-08T22:55:38.089 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status -f json 2022-03-08T22:55:38.745 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:55:38.746 INFO:teuthology.orchestra.run.smithi186.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:38.746 INFO:teuthology.orchestra.run.smithi186.stderr:pg stat 2022-03-08T22:55:38.746 INFO:teuthology.orchestra.run.smithi186.stderr:pg getmap 2022-03-08T22:55:38.747 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:38.747 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:38.747 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_pools_json 2022-03-08T22:55:38.747 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-pool [...] 2022-03-08T22:55:38.747 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:38.748 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:38.748 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls [] [...] 2022-03-08T22:55:38.748 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:38.748 INFO:teuthology.orchestra.run.smithi186.stderr:Error EINVAL: invalid command 2022-03-08T22:55:38.749 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 22 2022-03-08T22:55:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:39.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.054148 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:39.542 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.542769 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:55:39.848 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.848974 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:55:39.848 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.848991 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:55:39.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.848997 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:55:39.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.879441 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:55:39.884 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.883142 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:55:39.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:39.883527 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:55:40.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:40.090381 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:55:40.606 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:40.607422 I | op-mon: start running mons 2022-03-08T22:55:40.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:40.675705 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:55:40.705 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:40.706075 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T22:55:40.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:40.837718 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:55:40.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:40.837870 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:55:41.003 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:41.004282 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T22:55:41.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:41.854129 I | op-mon: targeting the mon count 1 2022-03-08T22:55:41.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:41.888538 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:42.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.175446 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:42.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.175481 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:42.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.396738 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:42.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.396761 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:42.617 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.617363 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:42.617 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.617402 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:42.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.837802 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:42.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:42.837831 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:55:43.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.061078 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:55:43.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.061098 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:55:43.286 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.286660 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:55:43.286 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.286688 I | op-mon: checking for basic quorum with existing mons 2022-03-08T22:55:43.442 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.443334 I | op-mon: mon "a" endpoint is [v2:10.107.60.203:3300,v1:10.107.60.203:6789] 2022-03-08T22:55:43.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.456223 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T22:55:43.823 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.823758 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:55:43.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:43.824079 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:55:44.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.223960 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T22:55:44.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.231245 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T22:55:44.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.231271 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:55:44.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.349556 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:44.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.414436 I | op-mon: mons running: [a] 2022-03-08T22:55:44.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.637735 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:44.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.637759 I | op-mon: mons created: 1 2022-03-08T22:55:44.925 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.926205 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:55:44.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:44.947117 I | op-mon: mons running: [a] 2022-03-08T22:55:45.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:45.154943 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:45.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:45.154973 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:55:45.383 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:45.383495 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:55:45.623 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:45.623033 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:55:45.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:45.892657 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:55:46.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.227840 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:55:46.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.227870 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:55:46.298 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.298947 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T22:55:46.613 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.613878 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:55:46.848 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.848964 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:55:46.848 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.849003 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:55:46.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:46.849012 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:55:47.109 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:47.110179 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:55:47.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:47.131745 I | op-mgr: start running mgr 2022-03-08T22:55:47.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:47.131774 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:55:47.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:47.645450 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T22:55:47.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:47.660095 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T22:55:47.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:47.662628 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:55:48.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.432317 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:55:48.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.432345 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T22:55:48.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.432354 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:55:48.456 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.456407 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:55:48.611 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.612447 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T22:55:48.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.612542 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:55:48.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:48.612574 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:55:48.752 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status -f json 2022-03-08T22:55:49.214 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:55:49.215 INFO:teuthology.orchestra.run.smithi186.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:49.215 INFO:teuthology.orchestra.run.smithi186.stderr:pg stat 2022-03-08T22:55:49.216 INFO:teuthology.orchestra.run.smithi186.stderr:pg getmap 2022-03-08T22:55:49.216 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:49.216 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:49.216 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_pools_json 2022-03-08T22:55:49.217 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-pool [...] 2022-03-08T22:55:49.217 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:49.217 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:49.217 INFO:teuthology.orchestra.run.smithi186.stderr:pg ls [] [...] 2022-03-08T22:55:49.218 INFO:teuthology.orchestra.run.smithi186.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:49.218 INFO:teuthology.orchestra.run.smithi186.stderr:Error EINVAL: invalid command 2022-03-08T22:55:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:49.220 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 22 2022-03-08T22:55:49.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.273214 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T22:55:49.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.273241 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:55:49.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.330058 I | op-mgr: successful modules: prometheus 2022-03-08T22:55:49.332 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.330869 I | op-mgr: successful modules: dashboard 2022-03-08T22:55:49.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.336656 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:55:49.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.615007 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:49.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:49.740565 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:50.675 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:50.676055 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T22:55:51.628 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:51.629558 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T22:55:55.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:55.040607 I | op-mgr: successful modules: balancer 2022-03-08T22:55:55.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:55.089615 I | op-mgr: successful modules: balancer 2022-03-08T22:55:56.115 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:56.115980 I | cephclient: action set rook backend succeeded on attempt 1 2022-03-08T22:55:56.115 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:56.115998 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:55:56.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:56.998018 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-08T22:55:56.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:55:56.998048 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:55:59.221 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status -f json 2022-03-08T22:55:59.774 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:55:59.775 INFO:teuthology.orchestra.run.smithi186.stdout:{"available": true, "backend": "rook"} 2022-03-08T22:55:59.799 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T22:55:59.799 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T22:56:00.449 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch apply osd --all-available-devices 2022-03-08T22:56:01.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:01.117954 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T22:56:01.117 INFO:teuthology.orchestra.run.smithi186.stdout:['Success'] 2022-03-08T22:56:01.120 INFO:tasks.rook.operator.smithi186.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T22:56:01.121 INFO:tasks.rook.operator.smithi186.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T22:56:01.121 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-08T22:56:01.121 INFO:tasks.rook.operator.smithi186.stdout: Config: nil, 2022-03-08T22:56:01.122 INFO:tasks.rook.operator.smithi186.stdout: Selection: {}, 2022-03-08T22:56:01.122 INFO:tasks.rook.operator.smithi186.stdout:- StorageClassDeviceSets: nil, 2022-03-08T22:56:01.122 INFO:tasks.rook.operator.smithi186.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T22:56:01.122 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-08T22:56:01.123 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-1", 2022-03-08T22:56:01.123 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-08T22:56:01.123 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.123 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-08T22:56:01.124 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-08T22:56:01.124 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-2", 2022-03-08T22:56:01.124 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-08T22:56:01.124 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.125 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-08T22:56:01.125 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-08T22:56:01.125 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-3", 2022-03-08T22:56:01.125 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-08T22:56:01.125 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.126 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-08T22:56:01.126 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-08T22:56:01.126 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-4", 2022-03-08T22:56:01.126 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-08T22:56:01.127 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.127 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-08T22:56:01.127 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-08T22:56:01.127 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-08T22:56:01.128 INFO:tasks.rook.operator.smithi186.stdout: Annotations: nil, 2022-03-08T22:56:01.128 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-08T22:56:01.128 INFO:tasks.rook.operator.smithi186.stdout: ... // 20 identical fields 2022-03-08T22:56:01.129 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-08T22:56:01.129 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:01.118099 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:56:01.141 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-08T22:56:01.142 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:56:01.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:01.141957 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:56:01.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:01.163538 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:56:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:56:01.790 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:01.791 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:06.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:06.926864 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:56:06.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:06.926892 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:56:06.927 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:06.926902 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:56:06.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:06.932054 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:56:07.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.292857 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:07.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.293070 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:07.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.730646 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:56:07.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.828393 I | op-mon: start running mons 2022-03-08T22:56:07.848 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.849693 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:56:07.863 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.864240 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T22:56:07.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.871627 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:07.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:07.871801 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:08.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:08.968707 I | op-mon: targeting the mon count 1 2022-03-08T22:56:09.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:09.163399 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T22:56:10.163 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:10.163736 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:56:10.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:10.170847 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:56:10.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:10.183931 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:56:10.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:10.215724 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T22:56:10.219 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:10.221426 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T22:56:11.792 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:56:12.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:12.229926 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T22:56:12.354 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:56:12.371 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:12.371 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:18.301 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:18.302887 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:56:18.302 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:18.302910 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:56:18.302 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:18.302921 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:56:18.740 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:18.741309 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:56:18.746 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:18.747505 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:18.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:18.750165 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:19.301 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:19.302205 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:56:19.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:19.379105 I | op-mon: start running mons 2022-03-08T22:56:19.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:19.404973 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T22:56:19.511 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:19.513313 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T22:56:19.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:19.535825 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:19.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:19.536082 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:20.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:20.368267 I | op-mon: targeting the mon count 1 2022-03-08T22:56:20.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:20.372610 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:56:20.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:20.754086 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:56:20.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:20.754124 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:56:21.090 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:21.092148 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:56:21.091 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:21.092191 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:56:21.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:21.431190 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:56:21.430 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:21.431229 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:56:21.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:21.774571 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:56:21.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:21.774603 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:56:22.106 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.107833 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:56:22.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.107866 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:56:22.372 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:56:22.458 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.459849 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:56:22.459 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.459873 I | op-mon: checking for basic quorum with existing mons 2022-03-08T22:56:22.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.653826 I | op-mon: mon "a" endpoint is [v2:10.107.60.203:3300,v1:10.107.60.203:6789] 2022-03-08T22:56:22.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.694674 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T22:56:22.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.742614 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:22.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.742814 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:22.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.897786 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T22:56:22.904 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.906219 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T22:56:22.904 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:22.906239 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:56:22.925 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:56:22.944 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:22.944 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:23.067 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:23.068771 I | op-mon: mons running: [a] 2022-03-08T22:56:23.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:23.484549 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:56:23.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:23.484579 I | op-mon: mons created: 1 2022-03-08T22:56:23.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:23.889948 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:56:23.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:23.895653 I | op-mon: mons running: [a] 2022-03-08T22:56:24.287 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:24.288353 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:56:24.287 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:24.288393 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:56:24.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:24.698234 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:56:25.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:25.105495 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:56:25.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:25.524350 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:56:25.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:25.961234 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:56:25.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:25.961255 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:56:26.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:26.402920 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:56:26.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:26.808212 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:56:26.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:26.808248 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:56:26.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:26.808256 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:56:27.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.229353 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:56:27.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.257991 I | op-mgr: start running mgr 2022-03-08T22:56:27.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.258038 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:56:27.729 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.730883 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T22:56:27.735 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.736852 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T22:56:27.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.739120 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:56:27.921 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.923096 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:56:27.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.923195 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:56:27.947 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.949014 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:56:27.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:27.951717 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-1" 2022-03-08T22:56:28.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.154662 I | op-osd: successfully provisioned PVC "smithi186-lv-1-data-0b8dfr" 2022-03-08T22:56:28.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.154914 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-2" 2022-03-08T22:56:28.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.381189 I | op-osd: successfully provisioned PVC "smithi186-lv-2-data-06cpzl" 2022-03-08T22:56:28.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.381214 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-3" 2022-03-08T22:56:28.551 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.552240 I | op-osd: successfully provisioned PVC "smithi186-lv-3-data-04q5qg" 2022-03-08T22:56:28.551 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.552291 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-4" 2022-03-08T22:56:28.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.802194 I | op-osd: successfully provisioned PVC "smithi186-lv-4-data-0r9vqb" 2022-03-08T22:56:28.805 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:28.807057 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0b8dfr" 2022-03-08T22:56:29.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.176267 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-1-data-0b8dfr" 2022-03-08T22:56:29.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.176301 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-06cpzl" 2022-03-08T22:56:29.430 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.432108 I | op-mgr: successful modules: dashboard 2022-03-08T22:56:29.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.438839 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:56:29.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.439652 I | op-mgr: successful modules: prometheus 2022-03-08T22:56:29.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.578909 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-2-data-06cpzl" 2022-03-08T22:56:29.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.578939 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-04q5qg" 2022-03-08T22:56:29.785 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.787511 I | op-mgr: successful modules: balancer 2022-03-08T22:56:29.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.972576 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-3-data-04q5qg" 2022-03-08T22:56:29.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:29.972607 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0r9vqb" 2022-03-08T22:56:30.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:30.831793 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-4-data-0r9vqb" 2022-03-08T22:56:30.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:30.831817 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:56:30.831 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:30.831820 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:56:31.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:31.140496 I | op-osd: OSD orchestration status for PVC smithi186-lv-1-data-0b8dfr is "starting" 2022-03-08T22:56:31.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:31.140723 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-06cpzl is "starting" 2022-03-08T22:56:31.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:31.140831 I | op-osd: OSD orchestration status for PVC smithi186-lv-3-data-04q5qg is "starting" 2022-03-08T22:56:31.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:31.140963 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0r9vqb is "starting" 2022-03-08T22:56:31.700 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:31.702350 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:56:32.945 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:56:33.470 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:56:33.486 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:33.487 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:43.488 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:56:44.060 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:56:44.081 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:44.082 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:45.220 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:45.222404 I | op-osd: OSD orchestration status for node smithi186-lv-2-data-06cpzl is "orchestrating" 2022-03-08T22:56:46.113 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:46.114844 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-06cpzl is "orchestrating" 2022-03-08T22:56:47.668 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:47.670684 I | op-osd: OSD orchestration status for node smithi186-lv-3-data-04q5qg is "orchestrating" 2022-03-08T22:56:48.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:48.478806 I | op-osd: OSD orchestration status for PVC smithi186-lv-3-data-04q5qg is "orchestrating" 2022-03-08T22:56:53.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:53.954277 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-06cpzl is "completed" 2022-03-08T22:56:53.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:53.954735 I | op-osd: creating OSD 0 on PVC "smithi186-lv-2-data-06cpzl" 2022-03-08T22:56:53.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:53.954778 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-06cpzl" 2022-03-08T22:56:54.083 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:56:55.279 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:56:55.296 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":8,"num_osds":2,"num_up_osds":0,"osd_up_since":0,"num_in_osds":2,"osd_in_since":1646780210,"num_remapped_pgs":0} 2022-03-08T22:56:55.297 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:55.394 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:55.393276 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:56:55.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:55.547671 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:56:55.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:55.550483 I | op-osd: OSD orchestration status for PVC smithi186-lv-3-data-04q5qg is "completed" 2022-03-08T22:56:55.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:55.550518 I | op-osd: creating OSD 1 on PVC "smithi186-lv-3-data-04q5qg" 2022-03-08T22:56:55.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:55.550531 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-04q5qg" 2022-03-08T22:56:55.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:55.595034 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:56:57.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:57.347062 I | op-osd: OSD orchestration status for node smithi186-lv-1-data-0b8dfr is "orchestrating" 2022-03-08T22:56:57.347 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:57.347364 I | op-osd: OSD orchestration status for node smithi186-lv-4-data-0r9vqb is "orchestrating" 2022-03-08T22:56:57.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:57.347386 I | op-osd: OSD orchestration status for PVC smithi186-lv-1-data-0b8dfr is "orchestrating" 2022-03-08T22:56:57.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:56:57.374891 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0r9vqb is "orchestrating" 2022-03-08T22:57:04.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:04.852909 I | op-osd: OSD orchestration status for PVC smithi186-lv-1-data-0b8dfr is "completed" 2022-03-08T22:57:04.851 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:04.852956 I | op-osd: creating OSD 2 on PVC "smithi186-lv-1-data-0b8dfr" 2022-03-08T22:57:04.851 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:04.852977 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0b8dfr" 2022-03-08T22:57:05.298 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:57:06.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:06.717553 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0r9vqb is "completed" 2022-03-08T22:57:06.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:06.717585 I | op-osd: creating OSD 3 on PVC "smithi186-lv-4-data-0r9vqb" 2022-03-08T22:57:06.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:06.717596 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0r9vqb" 2022-03-08T22:57:06.771 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780219,"num_remapped_pgs":0} 2022-03-08T22:57:06.790 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:57:09.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:09.634657 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T22:57:09.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:09.634678 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:57:16.791 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:57:18.161 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:18.177 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780219,"num_remapped_pgs":0} 2022-03-08T22:57:18.178 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:57:22.985 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 22:57:22.987148 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:57:28.179 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:57:29.301 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:29.326 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780219,"num_remapped_pgs":0} 2022-03-08T22:57:29.326 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:57:39.327 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:57:39.842 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:39.860 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":15,"num_osds":4,"num_up_osds":2,"osd_up_since":1646780259,"num_in_osds":4,"osd_in_since":1646780219,"num_remapped_pgs":0} 2022-03-08T22:57:39.860 INFO:tasks.rook: have 2/4 OSDs 2022-03-08T22:57:49.861 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd stat -f json 2022-03-08T22:57:50.355 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:50.372 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":22,"num_osds":4,"num_up_osds":4,"osd_up_since":1646780266,"num_in_osds":4,"osd_in_since":1646780219,"num_remapped_pgs":0} 2022-03-08T22:57:50.373 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T22:57:50.373 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- cat /etc/ceph/ceph.conf 2022-03-08T22:57:50.542 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2022-03-08T22:57:50.542 INFO:teuthology.orchestra.run.smithi186.stdout:mon_host = 10.107.60.203:6789 2022-03-08T22:57:50.543 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:50.543 INFO:teuthology.orchestra.run.smithi186.stdout:[client.admin] 2022-03-08T22:57:50.543 INFO:teuthology.orchestra.run.smithi186.stdout:keyring = /etc/ceph/keyring 2022-03-08T22:57:50.545 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- cat /etc/ceph/keyring 2022-03-08T22:57:50.711 INFO:teuthology.orchestra.run.smithi186.stdout:[client.admin] 2022-03-08T22:57:50.712 INFO:teuthology.orchestra.run.smithi186.stdout:key = AQCk3idirRnwIBAAbwgT8wuYLDNq6U+0f6hUzg== 2022-03-08T22:57:50.718 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2022-03-08T22:57:50.730 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:57:50.731 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T22:57:50.783 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:57:50.783 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/keyring 2022-03-08T22:57:50.792 INFO:tasks.rook:Setting up client nodes... 2022-03-08T22:57:50.793 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T22:57:51.429 INFO:teuthology.orchestra.run.smithi186.stdout:[client.a] 2022-03-08T22:57:51.430 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBv3ydi0h63FRAAKPSFNLti7T67BqVSLdnfig== 2022-03-08T22:57:51.449 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:57:51.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T22:57:51.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T22:57:51.466 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd dump --format=json 2022-03-08T22:57:51.965 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:51.966 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":23,"fsid":"7c67c3a2-0ced-49bf-85a1-f4cdf61e1f29","created":"2022-03-08T22:55:16.995738+0000","modified":"2022-03-08T22:57:50.512169+0000","last_up_change":"2022-03-08T22:57:46.505345+0000","last_in_change":"2022-03-08T22:56:59.090070+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T22:57:47.585768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"063baec2-7923-488d-880e-3be11255fa61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6800","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6801","nonce":484646060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6802","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6803","nonce":484646060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6806","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6807","nonce":484646060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6804","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6805","nonce":484646060}]},"public_addr":"10.253.202.151:6801/484646060","cluster_addr":"10.253.202.151:6803/484646060","heartbeat_back_addr":"10.253.202.151:6807/484646060","heartbeat_front_addr":"10.253.202.151:6805/484646060","state":["exists","up"]},{"osd":1,"uuid":"d2bf2c7d-a243-4aa4-969c-0b8cdcf7ef5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6800","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6801","nonce":3323352078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6802","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6803","nonce":3323352078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6806","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6807","nonce":3323352078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6804","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6805","nonce":3323352078}]},"public_addr":"10.253.202.152:6801/3323352078","cluster_addr":"10.253.202.152:6803/3323352078","heartbeat_back_addr":"10.253.202.152:6807/3323352078","heartbeat_front_addr":"10.253.202.152:6805/3323352078","state":["exists","up"]},{"osd":2,"uuid":"2e1d95c9-8658-4d2e-8f21-d515c0e7e282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6800","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6801","nonce":591011497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6802","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6803","nonce":591011497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6806","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6807","nonce":591011497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6804","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6805","nonce":591011497}]},"public_addr":"10.253.202.153:6801/591011497","cluster_addr":"10.253.202.153:6803/591011497","heartbeat_back_addr":"10.253.202.153:6807/591011497","heartbeat_front_addr":"10.253.202.153:6805/591011497","state":["exists","up"]},{"osd":3,"uuid":"ac038d02-07e3-48dd-b727-d3f225d727a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6800","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6801","nonce":409296601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6802","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6803","nonce":409296601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6806","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6807","nonce":409296601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6804","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6805","nonce":409296601}]},"public_addr":"10.253.202.154:6801/409296601","cluster_addr":"10.253.202.154:6803/409296601","heartbeat_back_addr":"10.253.202.154:6807/409296601","heartbeat_front_addr":"10.253.202.154:6805/409296601","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:36.808380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:34.733006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:43.123135+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:44.271948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T22:57:51.985 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T22:57:47.585768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T22:57:51.985 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd pool get .mgr pg_num 2022-03-08T22:57:52.462 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2022-03-08T22:57:52.480 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T22:57:52.481 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T22:57:52.481 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph mgr dump --format=json 2022-03-08T22:57:53.025 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:53.045 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":14,"active_gid":4192,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.142:6800","nonce":3340138347},{"type":"v1","addr":"10.253.202.142:6801","nonce":3340138347}]},"active_addr":"10.253.202.142:6801/3340138347","active_change":"2022-03-08T22:55:52.890650+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.253.202.142:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.253.202.142:0","nonce":1731639724}]},{"addrvec":[{"type":"v2","addr":"10.253.202.142:0","nonce":117816250}]},{"addrvec":[{"type":"v2","addr":"10.253.202.142:0","nonce":3639192472}]},{"addrvec":[{"type":"v2","addr":"10.253.202.142:0","nonce":4150917290}]}]}} 2022-03-08T22:57:53.048 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T22:57:53.048 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T22:57:53.049 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd dump --format=json 2022-03-08T22:57:53.563 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:53.563 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":23,"fsid":"7c67c3a2-0ced-49bf-85a1-f4cdf61e1f29","created":"2022-03-08T22:55:16.995738+0000","modified":"2022-03-08T22:57:50.512169+0000","last_up_change":"2022-03-08T22:57:46.505345+0000","last_in_change":"2022-03-08T22:56:59.090070+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T22:57:47.585768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"063baec2-7923-488d-880e-3be11255fa61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6800","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6801","nonce":484646060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6802","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6803","nonce":484646060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6806","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6807","nonce":484646060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6804","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6805","nonce":484646060}]},"public_addr":"10.253.202.151:6801/484646060","cluster_addr":"10.253.202.151:6803/484646060","heartbeat_back_addr":"10.253.202.151:6807/484646060","heartbeat_front_addr":"10.253.202.151:6805/484646060","state":["exists","up"]},{"osd":1,"uuid":"d2bf2c7d-a243-4aa4-969c-0b8cdcf7ef5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6800","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6801","nonce":3323352078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6802","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6803","nonce":3323352078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6806","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6807","nonce":3323352078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6804","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6805","nonce":3323352078}]},"public_addr":"10.253.202.152:6801/3323352078","cluster_addr":"10.253.202.152:6803/3323352078","heartbeat_back_addr":"10.253.202.152:6807/3323352078","heartbeat_front_addr":"10.253.202.152:6805/3323352078","state":["exists","up"]},{"osd":2,"uuid":"2e1d95c9-8658-4d2e-8f21-d515c0e7e282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6800","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6801","nonce":591011497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6802","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6803","nonce":591011497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6806","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6807","nonce":591011497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6804","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6805","nonce":591011497}]},"public_addr":"10.253.202.153:6801/591011497","cluster_addr":"10.253.202.153:6803/591011497","heartbeat_back_addr":"10.253.202.153:6807/591011497","heartbeat_front_addr":"10.253.202.153:6805/591011497","state":["exists","up"]},{"osd":3,"uuid":"ac038d02-07e3-48dd-b727-d3f225d727a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6800","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6801","nonce":409296601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6802","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6803","nonce":409296601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6806","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6807","nonce":409296601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6804","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6805","nonce":409296601}]},"public_addr":"10.253.202.154:6801/409296601","cluster_addr":"10.253.202.154:6803/409296601","heartbeat_back_addr":"10.253.202.154:6807/409296601","heartbeat_front_addr":"10.253.202.154:6805/409296601","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:36.808380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:34.733006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:43.123135+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:44.271948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T22:57:53.586 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T22:57:53.587 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd dump --format=json 2022-03-08T22:57:54.057 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:54.057 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":23,"fsid":"7c67c3a2-0ced-49bf-85a1-f4cdf61e1f29","created":"2022-03-08T22:55:16.995738+0000","modified":"2022-03-08T22:57:50.512169+0000","last_up_change":"2022-03-08T22:57:46.505345+0000","last_in_change":"2022-03-08T22:56:59.090070+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T22:57:47.585768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"063baec2-7923-488d-880e-3be11255fa61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6800","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6801","nonce":484646060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6802","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6803","nonce":484646060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6806","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6807","nonce":484646060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.151:6804","nonce":484646060},{"type":"v1","addr":"10.253.202.151:6805","nonce":484646060}]},"public_addr":"10.253.202.151:6801/484646060","cluster_addr":"10.253.202.151:6803/484646060","heartbeat_back_addr":"10.253.202.151:6807/484646060","heartbeat_front_addr":"10.253.202.151:6805/484646060","state":["exists","up"]},{"osd":1,"uuid":"d2bf2c7d-a243-4aa4-969c-0b8cdcf7ef5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6800","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6801","nonce":3323352078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6802","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6803","nonce":3323352078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6806","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6807","nonce":3323352078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6804","nonce":3323352078},{"type":"v1","addr":"10.253.202.152:6805","nonce":3323352078}]},"public_addr":"10.253.202.152:6801/3323352078","cluster_addr":"10.253.202.152:6803/3323352078","heartbeat_back_addr":"10.253.202.152:6807/3323352078","heartbeat_front_addr":"10.253.202.152:6805/3323352078","state":["exists","up"]},{"osd":2,"uuid":"2e1d95c9-8658-4d2e-8f21-d515c0e7e282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6800","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6801","nonce":591011497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6802","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6803","nonce":591011497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6806","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6807","nonce":591011497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6804","nonce":591011497},{"type":"v1","addr":"10.253.202.153:6805","nonce":591011497}]},"public_addr":"10.253.202.153:6801/591011497","cluster_addr":"10.253.202.153:6803/591011497","heartbeat_back_addr":"10.253.202.153:6807/591011497","heartbeat_front_addr":"10.253.202.153:6805/591011497","state":["exists","up"]},{"osd":3,"uuid":"ac038d02-07e3-48dd-b727-d3f225d727a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6800","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6801","nonce":409296601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6802","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6803","nonce":409296601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6806","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6807","nonce":409296601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6804","nonce":409296601},{"type":"v1","addr":"10.253.202.154:6805","nonce":409296601}]},"public_addr":"10.253.202.154:6801/409296601","cluster_addr":"10.253.202.154:6803/409296601","heartbeat_back_addr":"10.253.202.154:6807/409296601","heartbeat_front_addr":"10.253.202.154:6805/409296601","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:36.808380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:34.733006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:43.123135+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:44.271948+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T22:57:54.080 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph tell osd.0 flush_pg_stats 2022-03-08T22:57:54.081 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph tell osd.1 flush_pg_stats 2022-03-08T22:57:54.081 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph tell osd.2 flush_pg_stats 2022-03-08T22:57:54.082 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph tell osd.3 flush_pg_stats 2022-03-08T22:57:54.498 INFO:teuthology.orchestra.run.smithi186.stdout:81604378627 2022-03-08T22:57:54.498 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.2 2022-03-08T22:57:54.502 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2022-03-08T22:57:54.503 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.1 2022-03-08T22:57:54.595 INFO:teuthology.orchestra.run.smithi186.stdout:85899345924 2022-03-08T22:57:54.596 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.3 2022-03-08T22:57:54.645 INFO:teuthology.orchestra.run.smithi186.stdout:64424509445 2022-03-08T22:57:54.646 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.0 2022-03-08T22:57:55.053 INFO:teuthology.orchestra.run.smithi186.stdout:81604378626 2022-03-08T22:57:55.072 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378627 got 81604378626 for osd.2 2022-03-08T22:57:55.075 INFO:teuthology.orchestra.run.smithi186.stdout:55834574854 2022-03-08T22:57:55.093 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2022-03-08T22:57:55.119 INFO:teuthology.orchestra.run.smithi186.stdout:85899345923 2022-03-08T22:57:55.141 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345924 got 85899345923 for osd.3 2022-03-08T22:57:55.163 INFO:teuthology.orchestra.run.smithi186.stdout:64424509444 2022-03-08T22:57:55.180 INFO:tasks.rook.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.0 2022-03-08T22:57:56.074 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.2 2022-03-08T22:57:56.094 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.1 2022-03-08T22:57:56.142 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.3 2022-03-08T22:57:56.181 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd last-stat-seq osd.0 2022-03-08T22:57:56.686 INFO:teuthology.orchestra.run.smithi186.stdout:81604378627 2022-03-08T22:57:56.711 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378627 got 81604378627 for osd.2 2022-03-08T22:57:56.711 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:56.777 INFO:teuthology.orchestra.run.smithi186.stdout:85899345924 2022-03-08T22:57:56.779 INFO:teuthology.orchestra.run.smithi186.stdout:55834574855 2022-03-08T22:57:56.802 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2022-03-08T22:57:56.802 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:56.803 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345924 got 85899345924 for osd.3 2022-03-08T22:57:56.804 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:56.832 INFO:teuthology.orchestra.run.smithi186.stdout:64424509445 2022-03-08T22:57:56.853 INFO:tasks.rook.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.0 2022-03-08T22:57:56.854 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:56.854 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T22:57:56.855 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph pg dump --format=json 2022-03-08T22:57:57.381 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:57.384 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2022-03-08T22:57:57.400 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2022-03-08T22:57:57.066545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23484,"kb_used_data":2080,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":374957124,"statfs":{"total":383980142592,"available":383956094976,"internally_reserved":0,"allocated":2129920,"data_stored":1532642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000909"},"pg_stats":[{"pgid":"1.0","version":"23'97","reported_seq":137,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-03-08T22:57:54.083854+0000","last_change":"2022-03-08T22:57:49.864592+0000","last_active":"2022-03-08T22:57:54.083854+0000","last_peered":"2022-03-08T22:57:54.083854+0000","last_clean":"2022-03-08T22:57:54.083854+0000","last_became_active":"2022-03-08T22:57:49.862705+0000","last_became_peered":"2022-03-08T22:57:49.862705+0000","last_unstale":"2022-03-08T22:57:54.083854+0000","last_undegraded":"2022-03-08T22:57:54.083854+0000","last_fullsized":"2022-03-08T22:57:54.083854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T22:57:47.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T22:57:47.692305+0000","last_clean_scrub_stamp":"2022-03-08T22:57:47.692305+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T06:36:31.722245+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":192512,"data_stored":47973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5904,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739248,"statfs":{"total":95995035648,"available":95988989952,"internally_reserved":0,"allocated":663552,"data_stored":514929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6164,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738988,"statfs":{"total":95995035648,"available":95988723712,"internally_reserved":0,"allocated":606208,"data_stored":454150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6356,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":667648,"data_stored":515590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T22:57:57.401 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph pg dump --format=json 2022-03-08T22:57:57.897 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:57.900 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2022-03-08T22:57:57.917 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2022-03-08T22:57:57.066545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23484,"kb_used_data":2080,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":374957124,"statfs":{"total":383980142592,"available":383956094976,"internally_reserved":0,"allocated":2129920,"data_stored":1532642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000909"},"pg_stats":[{"pgid":"1.0","version":"23'97","reported_seq":137,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-03-08T22:57:54.083854+0000","last_change":"2022-03-08T22:57:49.864592+0000","last_active":"2022-03-08T22:57:54.083854+0000","last_peered":"2022-03-08T22:57:54.083854+0000","last_clean":"2022-03-08T22:57:54.083854+0000","last_became_active":"2022-03-08T22:57:49.862705+0000","last_became_peered":"2022-03-08T22:57:49.862705+0000","last_unstale":"2022-03-08T22:57:54.083854+0000","last_undegraded":"2022-03-08T22:57:54.083854+0000","last_fullsized":"2022-03-08T22:57:54.083854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T22:57:47.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T22:57:47.692305+0000","last_clean_scrub_stamp":"2022-03-08T22:57:47.692305+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T06:36:31.722245+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":192512,"data_stored":47973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5904,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739248,"statfs":{"total":95995035648,"available":95988989952,"internally_reserved":0,"allocated":663552,"data_stored":514929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6164,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738988,"statfs":{"total":95995035648,"available":95988723712,"internally_reserved":0,"allocated":606208,"data_stored":454150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6356,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":667648,"data_stored":515590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T22:57:57.918 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T22:57:57.919 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T22:57:57.919 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T22:57:57.919 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph health --format=json 2022-03-08T22:57:58.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:58.431 INFO:teuthology.orchestra.run.smithi186.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T22:57:58.456 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T22:57:58.457 INFO:tasks.rook:Rook complete, yielding 2022-03-08T22:57:58.458 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T22:57:58.472 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph -s 2022-03-08T22:57:59.037 INFO:teuthology.orchestra.run.smithi186.stdout: cluster: 2022-03-08T22:57:59.037 INFO:teuthology.orchestra.run.smithi186.stdout: id: 7c67c3a2-0ced-49bf-85a1-f4cdf61e1f29 2022-03-08T22:57:59.038 INFO:teuthology.orchestra.run.smithi186.stdout: health: HEALTH_OK 2022-03-08T22:57:59.038 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:59.038 INFO:teuthology.orchestra.run.smithi186.stdout: services: 2022-03-08T22:57:59.038 INFO:teuthology.orchestra.run.smithi186.stdout: mon: 1 daemons, quorum a (age 2m) 2022-03-08T22:57:59.039 INFO:teuthology.orchestra.run.smithi186.stdout: mgr: a(active, since 2m) 2022-03-08T22:57:59.039 INFO:teuthology.orchestra.run.smithi186.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 59s) 2022-03-08T22:57:59.039 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:59.039 INFO:teuthology.orchestra.run.smithi186.stdout: data: 2022-03-08T22:57:59.040 INFO:teuthology.orchestra.run.smithi186.stdout: pools: 1 pools, 1 pgs 2022-03-08T22:57:59.040 INFO:teuthology.orchestra.run.smithi186.stdout: objects: 2 objects, 577 KiB 2022-03-08T22:57:59.040 INFO:teuthology.orchestra.run.smithi186.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-03-08T22:57:59.040 INFO:teuthology.orchestra.run.smithi186.stdout: pgs: 1 active+clean 2022-03-08T22:57:59.041 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:59.041 INFO:teuthology.orchestra.run.smithi186.stdout: io: 2022-03-08T22:57:59.041 INFO:teuthology.orchestra.run.smithi186.stdout: client: 2.2 KiB/s rd, 65 KiB/s wr, 2 op/s rd, 3 op/s wr 2022-03-08T22:57:59.041 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T22:57:59.057 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status 2022-03-08T22:57:59.639 INFO:teuthology.orchestra.run.smithi186.stdout:Backend: rook 2022-03-08T22:57:59.639 INFO:teuthology.orchestra.run.smithi186.stdout:Available: Yes 2022-03-08T22:57:59.659 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ps 2022-03-08T22:58:00.177 INFO:teuthology.orchestra.run.smithi186.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T22:58:00.178 INFO:teuthology.orchestra.run.smithi186.stdout:crashcollector.crash smithi186 running (65s) 0s ago 65s - - 1c5288aef625 2022-03-08T22:58:00.178 INFO:teuthology.orchestra.run.smithi186.stdout:mgr.a smithi186 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-08T22:58:00.178 INFO:teuthology.orchestra.run.smithi186.stdout:mon.a smithi186 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-08T22:58:00.179 INFO:teuthology.orchestra.run.smithi186.stdout:osd.0 smithi186 running (65s) 0s ago 65s - - 1c5288aef625 2022-03-08T22:58:00.179 INFO:teuthology.orchestra.run.smithi186.stdout:osd.1 smithi186 running (63s) 0s ago 63s - - 1c5288aef625 2022-03-08T22:58:00.179 INFO:teuthology.orchestra.run.smithi186.stdout:osd.2 smithi186 running (54s) 0s ago 54s - - 1c5288aef625 2022-03-08T22:58:00.179 INFO:teuthology.orchestra.run.smithi186.stdout:osd.3 smithi186 running (51s) 0s ago 52s - - 1c5288aef625 2022-03-08T22:58:00.200 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls 2022-03-08T22:58:00.777 INFO:teuthology.orchestra.run.smithi186.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T22:58:00.778 INFO:teuthology.orchestra.run.smithi186.stdout:crash 1/1 0s ago 65s * 2022-03-08T22:58:00.778 INFO:teuthology.orchestra.run.smithi186.stdout:mgr 1/1 0s ago 2m count:1 2022-03-08T22:58:00.778 INFO:teuthology.orchestra.run.smithi186.stdout:mon 1/1 0s ago 3m count:1 2022-03-08T22:58:00.779 INFO:teuthology.orchestra.run.smithi186.stdout:osd 8 0s ago 65s 2022-03-08T22:58:00.779 INFO:teuthology.orchestra.run.smithi186.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T22:58:00.801 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch device ls 2022-03-08T22:58:01.360 INFO:teuthology.orchestra.run.smithi186.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T22:58:01.361 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T22:58:01.361 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T22:58:01.361 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T22:58:01.362 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T22:58:01.380 INFO:teuthology.run_tasks:Running task install... 2022-03-08T22:58:01.390 DEBUG:teuthology.task.install:project ceph 2022-03-08T22:58:01.390 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4'}} 2022-03-08T22:58:01.391 DEBUG:teuthology.task.install:config {'host.a': None, 'flavor': 'default', 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4'} 2022-03-08T22:58:01.391 INFO:teuthology.task.install:Using flavor: default 2022-03-08T22:58:01.398 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-08T22:58:01.398 INFO:teuthology.task.install:extra packages: [] 2022-03-08T22:58:01.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-08T22:58:01.441 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-08T22:58:01.446 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:58:01.446 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-key list | grep Ceph 2022-03-08T22:58:01.581 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-08T22:58:01.637 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-08T22:58:01.638 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph.com (release key) 2022-03-08T22:58:01.638 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-08T22:58:01.683 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-08T22:58:01.689 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:58:01.689 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-08T22:58:01.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:58:01.858 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default/ 2022-03-08T22:58:01.858 INFO:teuthology.task.install.deb:Package version is 17.0.0-11002-g94cb3430-1focal 2022-03-08T22:58:01.977 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T22:58:01.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-08T22:58:01.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2022-03-08T22:58:02.183 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:58:02.238 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal InRelease 2022-03-08T22:58:02.262 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-08T22:58:02.286 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal Release.gpg 2022-03-08T22:58:02.302 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:58:02.309 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main i386 Packages [3,353 B] 2022-03-08T22:58:02.342 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2022-03-08T22:58:02.385 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:58:02.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:58:02.468 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:58:02.884 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 37.7 kB in 1s (58.0 kB/s) 2022-03-08T22:58:03.831 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:58:03.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11002-g94cb3430-1focal cephadm=17.0.0-11002-g94cb3430-1focal ceph-mds=17.0.0-11002-g94cb3430-1focal ceph-mgr=17.0.0-11002-g94cb3430-1focal ceph-common=17.0.0-11002-g94cb3430-1focal ceph-fuse=17.0.0-11002-g94cb3430-1focal ceph-test=17.0.0-11002-g94cb3430-1focal ceph-volume=17.0.0-11002-g94cb3430-1focal radosgw=17.0.0-11002-g94cb3430-1focal python3-rados=17.0.0-11002-g94cb3430-1focal python3-rgw=17.0.0-11002-g94cb3430-1focal python3-cephfs=17.0.0-11002-g94cb3430-1focal python3-rbd=17.0.0-11002-g94cb3430-1focal libcephfs2=17.0.0-11002-g94cb3430-1focal libcephfs-dev=17.0.0-11002-g94cb3430-1focal librados2=17.0.0-11002-g94cb3430-1focal librbd1=17.0.0-11002-g94cb3430-1focal rbd-fuse=17.0.0-11002-g94cb3430-1focal 2022-03-08T22:58:03.921 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-08T22:58:04.103 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-08T22:58:04.104 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-08T22:58:04.270 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2022-03-08T22:58:04.271 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-08T22:58:04.272 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-08T22:58:04.272 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2022-03-08T22:58:04.272 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-08T22:58:04.273 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2022-03-08T22:58:04.273 INFO:teuthology.orchestra.run.smithi186.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T22:58:04.274 INFO:teuthology.orchestra.run.smithi186.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T22:58:04.274 INFO:teuthology.orchestra.run.smithi186.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T22:58:04.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T22:58:04.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-03-08T22:58:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T22:58:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T22:58:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T22:58:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T22:58:04.275 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T22:58:04.276 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T22:58:04.276 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-08T22:58:04.276 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-08T22:58:04.276 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-08T22:58:04.277 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-08T22:58:04.277 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-08T22:58:04.277 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket python3-webtest python3-werkzeug xmlstarlet 2022-03-08T22:58:04.278 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2022-03-08T22:58:04.278 INFO:teuthology.orchestra.run.smithi186.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-08T22:58:04.278 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-08T22:58:04.278 INFO:teuthology.orchestra.run.smithi186.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-08T22:58:04.279 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-08T22:58:04.279 INFO:teuthology.orchestra.run.smithi186.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-08T22:58:04.279 INFO:teuthology.orchestra.run.smithi186.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-08T22:58:04.279 INFO:teuthology.orchestra.run.smithi186.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-08T22:58:04.279 INFO:teuthology.orchestra.run.smithi186.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-08T22:58:04.280 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2022-03-08T22:58:04.280 INFO:teuthology.orchestra.run.smithi186.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-08T22:58:04.376 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-08T22:58:04.376 INFO:teuthology.orchestra.run.smithi186.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-08T22:58:04.377 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-08T22:58:04.378 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-08T22:58:04.378 INFO:teuthology.orchestra.run.smithi186.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T22:58:04.379 INFO:teuthology.orchestra.run.smithi186.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T22:58:04.379 INFO:teuthology.orchestra.run.smithi186.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T22:58:04.379 INFO:teuthology.orchestra.run.smithi186.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T22:58:04.380 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-03-08T22:58:04.380 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T22:58:04.380 INFO:teuthology.orchestra.run.smithi186.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T22:58:04.380 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T22:58:04.381 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T22:58:04.381 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T22:58:04.381 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T22:58:04.382 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-08T22:58:04.382 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-08T22:58:04.382 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-08T22:58:04.382 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-08T22:58:04.383 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-08T22:58:04.383 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-08T22:58:04.383 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse xmlstarlet 2022-03-08T22:58:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages will be upgraded: 2022-03-08T22:58:04.384 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 librbd1 2022-03-08T22:58:04.530 INFO:teuthology.orchestra.run.smithi186.stdout:2 upgraded, 95 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:58:04.531 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 181 MB of archives. 2022-03-08T22:58:04.531 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 845 MB of additional disk space will be used. 2022-03-08T22:58:04.531 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11002-g94cb3430-1focal [3,484 kB] 2022-03-08T22:58:04.570 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-08T22:58:04.744 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11002-g94cb3430-1focal [3,534 kB] 2022-03-08T22:58:04.852 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11002-g94cb3430-1focal [756 kB] 2022-03-08T22:58:04.876 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11002-g94cb3430-1focal [412 kB] 2022-03-08T22:58:04.890 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11002-g94cb3430-1focal [22.4 kB] 2022-03-08T22:58:04.891 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11002-g94cb3430-1focal [200 kB] 2022-03-08T22:58:04.897 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11002-g94cb3430-1focal [41.9 kB] 2022-03-08T22:58:04.898 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11002-g94cb3430-1focal [386 kB] 2022-03-08T22:58:04.914 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11002-g94cb3430-1focal [4,162 kB] 2022-03-08T22:58:04.940 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-08T22:58:04.949 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-08T22:58:04.952 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-08T22:58:04.963 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-08T22:58:05.048 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11002-g94cb3430-1focal [116 kB] 2022-03-08T22:58:05.051 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11002-g94cb3430-1focal [478 kB] 2022-03-08T22:58:05.067 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11002-g94cb3430-1focal [24.4 MB] 2022-03-08T22:58:05.123 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-08T22:58:05.129 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-08T22:58:05.163 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-08T22:58:05.170 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-08T22:58:05.174 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-08T22:58:05.195 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-08T22:58:05.197 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-08T22:58:05.198 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-08T22:58:05.199 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-08T22:58:05.205 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-08T22:58:05.287 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-08T22:58:05.287 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-08T22:58:05.290 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-08T22:58:05.291 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-08T22:58:05.294 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-08T22:58:05.297 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-08T22:58:05.298 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-08T22:58:05.312 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-08T22:58:05.312 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-08T22:58:05.371 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-08T22:58:05.372 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-08T22:58:05.453 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-08T22:58:05.459 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-08T22:58:05.460 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-08T22:58:05.461 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-08T22:58:05.465 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-08T22:58:05.466 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-08T22:58:05.467 INFO:teuthology.orchestra.run.smithi186.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-08T22:58:05.467 INFO:teuthology.orchestra.run.smithi186.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-08T22:58:05.865 INFO:teuthology.orchestra.run.smithi186.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-08T22:58:05.940 INFO:teuthology.orchestra.run.smithi186.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-08T22:58:06.033 INFO:teuthology.orchestra.run.smithi186.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-08T22:58:06.033 INFO:teuthology.orchestra.run.smithi186.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11002-g94cb3430-1focal [6,432 kB] 2022-03-08T22:58:06.045 INFO:teuthology.orchestra.run.smithi186.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-08T22:58:06.045 INFO:teuthology.orchestra.run.smithi186.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-08T22:58:06.046 INFO:teuthology.orchestra.run.smithi186.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-08T22:58:06.046 INFO:teuthology.orchestra.run.smithi186.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-08T22:58:06.047 INFO:teuthology.orchestra.run.smithi186.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-08T22:58:06.047 INFO:teuthology.orchestra.run.smithi186.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-08T22:58:06.047 INFO:teuthology.orchestra.run.smithi186.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-08T22:58:06.048 INFO:teuthology.orchestra.run.smithi186.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-08T22:58:06.115 INFO:teuthology.orchestra.run.smithi186.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-08T22:58:06.123 INFO:teuthology.orchestra.run.smithi186.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11002-g94cb3430-1focal [211 kB] 2022-03-08T22:58:06.128 INFO:teuthology.orchestra.run.smithi186.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11002-g94cb3430-1focal [151 kB] 2022-03-08T22:58:06.130 INFO:teuthology.orchestra.run.smithi186.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-08T22:58:06.134 INFO:teuthology.orchestra.run.smithi186.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-08T22:58:06.136 INFO:teuthology.orchestra.run.smithi186.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-08T22:58:06.136 INFO:teuthology.orchestra.run.smithi186.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11002-g94cb3430-1focal [1,575 kB] 2022-03-08T22:58:06.138 INFO:teuthology.orchestra.run.smithi186.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-08T22:58:06.192 INFO:teuthology.orchestra.run.smithi186.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-08T22:58:06.193 INFO:teuthology.orchestra.run.smithi186.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11002-g94cb3430-1focal [7,812 kB] 2022-03-08T22:58:06.193 INFO:teuthology.orchestra.run.smithi186.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-08T22:58:06.195 INFO:teuthology.orchestra.run.smithi186.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-08T22:58:06.198 INFO:teuthology.orchestra.run.smithi186.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-08T22:58:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-08T22:58:06.279 INFO:teuthology.orchestra.run.smithi186.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-08T22:58:06.280 INFO:teuthology.orchestra.run.smithi186.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-08T22:58:06.281 INFO:teuthology.orchestra.run.smithi186.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-08T22:58:06.298 INFO:teuthology.orchestra.run.smithi186.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-08T22:58:06.299 INFO:teuthology.orchestra.run.smithi186.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-08T22:58:06.299 INFO:teuthology.orchestra.run.smithi186.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-08T22:58:06.379 INFO:teuthology.orchestra.run.smithi186.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-08T22:58:06.382 INFO:teuthology.orchestra.run.smithi186.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-08T22:58:06.383 INFO:teuthology.orchestra.run.smithi186.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-08T22:58:06.384 INFO:teuthology.orchestra.run.smithi186.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-08T22:58:06.442 INFO:teuthology.orchestra.run.smithi186.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11002-g94cb3430-1focal [28.6 MB] 2022-03-08T22:58:06.461 INFO:teuthology.orchestra.run.smithi186.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-08T22:58:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-08T22:58:07.521 INFO:teuthology.orchestra.run.smithi186.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11002-g94cb3430-1focal [3,796 B] 2022-03-08T22:58:07.522 INFO:teuthology.orchestra.run.smithi186.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11002-g94cb3430-1focal [863 kB] 2022-03-08T22:58:07.528 INFO:teuthology.orchestra.run.smithi186.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11002-g94cb3430-1focal [2,397 kB] 2022-03-08T22:58:07.546 INFO:teuthology.orchestra.run.smithi186.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11002-g94cb3430-1focal [1,677 kB] 2022-03-08T22:58:07.579 INFO:teuthology.orchestra.run.smithi186.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11002-g94cb3430-1focal [7,888 kB] 2022-03-08T22:58:07.830 INFO:teuthology.orchestra.run.smithi186.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11002-g94cb3430-1focal [3,912 B] 2022-03-08T22:58:07.831 INFO:teuthology.orchestra.run.smithi186.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11002-g94cb3430-1focal [50.5 MB] 2022-03-08T22:58:09.679 INFO:teuthology.orchestra.run.smithi186.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11002-g94cb3430-1focal [219 kB] 2022-03-08T22:58:09.680 INFO:teuthology.orchestra.run.smithi186.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11002-g94cb3430-1focal [69.5 kB] 2022-03-08T22:58:09.795 INFO:teuthology.orchestra.run.smithi186.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11002-g94cb3430-1focal [21.7 kB] 2022-03-08T22:58:09.796 INFO:teuthology.orchestra.run.smithi186.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11002-g94cb3430-1focal [12.7 MB] 2022-03-08T22:58:10.093 INFO:teuthology.orchestra.run.smithi186.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11002-g94cb3430-1focal [78.9 kB] 2022-03-08T22:58:10.695 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 181 MB in 6s (31.7 MB/s) 2022-03-08T22:58:10.945 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-08T22:58:10.995 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-08T22:58:10.998 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-08T22:58:11.104 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T22:58:11.594 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../01-librbd1_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:11.738 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librbd1 (17.0.0-11002-g94cb3430-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T22:58:12.514 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../02-librados2_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:12.721 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librados2 (17.0.0-11002-g94cb3430-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T22:58:13.532 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs2. 2022-03-08T22:58:13.547 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:13.592 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:14.107 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rados. 2022-03-08T22:58:14.116 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../04-python3-rados_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:14.158 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rados (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:14.627 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-08T22:58:14.640 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:14.685 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-argparse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:15.034 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cephfs. 2022-03-08T22:58:15.043 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:15.084 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cephfs (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:15.533 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-08T22:58:15.541 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:15.603 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:16.044 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-prettytable. 2022-03-08T22:58:16.056 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-03-08T22:58:16.175 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-08T22:58:16.484 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rbd. 2022-03-08T22:58:16.496 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:16.551 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rbd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:16.997 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-08T22:58:17.009 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-08T22:58:17.038 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T22:58:17.545 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-08T22:58:17.556 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-08T22:58:17.629 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T22:58:17.985 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-08T22:58:17.997 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-08T22:58:18.037 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T22:58:18.554 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librgw2. 2022-03-08T22:58:18.567 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../13-librgw2_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:18.643 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librgw2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:19.373 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rgw. 2022-03-08T22:58:19.382 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:19.451 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:19.849 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liboath0. 2022-03-08T22:58:19.862 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-08T22:58:19.945 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-08T22:58:20.306 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libradosstriper1. 2022-03-08T22:58:20.321 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../16-libradosstriper1_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:20.366 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libradosstriper1 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:21.054 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-common. 2022-03-08T22:58:21.067 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../17-ceph-common_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:21.130 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:24.098 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-base. 2022-03-08T22:58:24.112 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../18-ceph-base_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:24.157 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-base (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:25.319 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cryptography. 2022-03-08T22:58:25.327 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-08T22:58:25.402 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T22:58:25.939 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-openssl. 2022-03-08T22:58:25.951 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-08T22:58:26.005 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-08T22:58:26.471 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-08T22:58:26.480 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:26.538 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:27.051 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-08T22:58:27.063 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-08T22:58:27.101 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T22:58:27.575 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-08T22:58:27.589 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-08T22:58:27.633 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-08T22:58:28.048 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jwt. 2022-03-08T22:58:28.060 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-08T22:58:28.107 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T22:58:28.635 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-logutils. 2022-03-08T22:58:28.643 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-03-08T22:58:28.712 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-08T22:58:29.013 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-08T22:58:29.027 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-08T22:58:29.096 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-08T22:58:29.440 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-mako. 2022-03-08T22:58:29.451 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-08T22:58:29.557 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T22:58:29.937 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-08T22:58:29.951 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-08T22:58:29.994 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-08T22:58:30.358 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-08T22:58:30.372 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-08T22:58:30.425 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T22:58:30.969 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webob. 2022-03-08T22:58:30.982 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-08T22:58:31.046 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-08T22:58:31.456 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-08T22:58:31.466 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-08T22:58:31.534 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T22:58:32.112 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bs4. 2022-03-08T22:58:32.124 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-03-08T22:58:32.177 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-08T22:58:32.652 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-waitress. 2022-03-08T22:58:32.661 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1_all.deb ... 2022-03-08T22:58:32.711 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-08T22:58:33.090 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempita. 2022-03-08T22:58:33.099 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-03-08T22:58:33.151 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-08T22:58:33.540 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-paste. 2022-03-08T22:58:33.552 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-08T22:58:33.629 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T22:58:34.016 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-08T22:58:34.029 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-08T22:58:34.092 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T22:58:34.411 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-08T22:58:34.426 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-08T22:58:34.502 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-08T22:58:34.854 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webtest. 2022-03-08T22:58:34.867 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-03-08T22:58:34.904 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-08T22:58:35.306 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pecan. 2022-03-08T22:58:35.314 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-08T22:58:35.356 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T22:58:36.481 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-08T22:58:36.489 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-08T22:58:36.809 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T22:58:37.247 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-08T22:58:37.260 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../41-libsqlite3-mod-ceph_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:37.305 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:37.733 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr. 2022-03-08T22:58:37.748 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../42-ceph-mgr_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:37.824 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:38.437 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mon. 2022-03-08T22:58:38.451 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../43-ceph-mon_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:38.495 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mon (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:39.725 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-osd. 2022-03-08T22:58:39.734 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../44-ceph-osd_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:39.783 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-osd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:43.255 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph. 2022-03-08T22:58:43.268 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../45-ceph_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:43.316 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:43.685 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-fuse. 2022-03-08T22:58:43.698 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../46-ceph-fuse_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:43.737 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:44.419 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mds. 2022-03-08T22:58:44.428 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../47-ceph-mds_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:44.478 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mds (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:45.122 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-08T22:58:45.137 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2022-03-08T22:58:45.198 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-08T22:58:45.599 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-underscore. 2022-03-08T22:58:45.607 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../49-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-08T22:58:45.657 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T22:58:46.014 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-08T22:58:46.026 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../50-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-08T22:58:46.088 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T22:58:46.534 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-routes. 2022-03-08T22:58:46.547 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../51-python3-routes_2.4.1-2_all.deb ... 2022-03-08T22:58:46.592 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-08T22:58:47.036 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-08T22:58:47.049 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../52-ceph-mgr-dashboard_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:47.088 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:47.755 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-decorator. 2022-03-08T22:58:47.770 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../53-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-08T22:58:47.827 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T22:58:48.247 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-08T22:58:48.261 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../54-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-08T22:58:48.305 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T22:58:48.758 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-scipy. 2022-03-08T22:58:48.768 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../55-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-08T22:58:48.813 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-08T22:58:50.731 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-08T22:58:50.741 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../56-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-08T22:58:50.789 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T22:58:51.552 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-joblib. 2022-03-08T22:58:51.565 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../57-python3-joblib_0.14.0-3_all.deb ... 2022-03-08T22:58:51.643 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-08T22:58:51.995 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn. 2022-03-08T22:58:52.008 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../58-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-08T22:58:52.053 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T22:58:53.156 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-08T22:58:53.167 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../59-ceph-mgr-diskprediction-local_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:53.206 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:54.610 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cachetools. 2022-03-08T22:58:54.622 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../60-python3-cachetools_4.0.0-1_all.deb ... 2022-03-08T22:58:54.689 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-08T22:58:55.164 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-08T22:58:55.176 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../61-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-08T22:58:55.229 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T22:58:55.576 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-08T22:58:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../62-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-08T22:58:55.633 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T22:58:56.104 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rsa. 2022-03-08T22:58:56.117 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../63-python3-rsa_4.0-3_all.deb ... 2022-03-08T22:58:56.189 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-08T22:58:56.755 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-google-auth. 2022-03-08T22:58:56.765 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../64-python3-google-auth_1.5.1-2_all.deb ... 2022-03-08T22:58:56.805 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-08T22:58:57.242 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-blinker. 2022-03-08T22:58:57.251 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../65-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-08T22:58:57.292 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T22:58:57.707 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-08T22:58:57.717 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../66-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-08T22:58:57.757 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T22:58:58.162 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-08T22:58:58.262 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../67-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-08T22:58:58.262 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T22:58:58.644 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-websocket. 2022-03-08T22:58:58.657 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../68-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-08T22:58:58.736 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T22:58:59.189 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-08T22:58:59.202 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../69-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-08T22:58:59.248 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T22:58:59.958 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-08T22:58:59.971 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../70-ceph-mgr-k8sevents_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:59:00.016 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:00.436 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-08T22:59:00.448 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../71-libonig5_6.9.4-1_amd64.deb ... 2022-03-08T22:59:00.486 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-08T22:59:01.032 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-08T22:59:01.045 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../72-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T22:59:01.124 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:59:01.448 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package jq. 2022-03-08T22:59:01.462 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../73-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T22:59:01.531 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:59:02.011 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-08T22:59:02.028 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../74-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-08T22:59:02.069 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T22:59:02.418 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package xmlstarlet. 2022-03-08T22:59:02.430 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../75-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-08T22:59:02.468 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-08T22:59:02.923 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-test. 2022-03-08T22:59:02.938 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../76-ceph-test_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:02.989 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-test (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:08.270 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-volume. 2022-03-08T22:59:08.283 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../77-ceph-volume_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:59:08.328 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-volume (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:08.788 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cephadm. 2022-03-08T22:59:08.803 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../78-cephadm_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:08.841 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cephadm (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:09.313 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-08T22:59:09.326 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../79-libcephfs-dev_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:09.384 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs-dev (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:09.825 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package nvme-cli. 2022-03-08T22:59:09.838 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../80-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-08T22:59:09.898 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T22:59:10.622 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-08T22:59:10.638 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../81-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-08T22:59:10.680 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T22:59:11.029 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-attr. 2022-03-08T22:59:11.043 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../82-python3-attr_19.3.0-2_all.deb ... 2022-03-08T22:59:11.091 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-08T22:59:11.637 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webencodings. 2022-03-08T22:59:11.648 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../83-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-08T22:59:11.687 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T22:59:12.039 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-html5lib. 2022-03-08T22:59:12.048 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../84-python3-html5lib_1.0.1-2_all.deb ... 2022-03-08T22:59:12.088 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-08T22:59:12.587 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-08T22:59:12.599 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../85-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-08T22:59:12.667 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T22:59:13.354 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-packaging. 2022-03-08T22:59:13.368 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../86-python3-packaging_20.3-1_all.deb ... 2022-03-08T22:59:13.572 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-08T22:59:14.649 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastescript. 2022-03-08T22:59:14.663 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../87-python3-pastescript_2.0.2-4_all.deb ... 2022-03-08T22:59:14.706 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-08T22:59:15.111 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pluggy. 2022-03-08T22:59:15.126 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../88-python3-pluggy_0.13.0-2_all.deb ... 2022-03-08T22:59:15.177 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-08T22:59:15.646 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-psutil. 2022-03-08T22:59:15.659 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../89-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-08T22:59:15.704 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T22:59:16.033 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-py. 2022-03-08T22:59:16.046 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../90-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-08T22:59:16.090 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T22:59:16.551 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-08T22:59:16.564 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../91-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-08T22:59:16.667 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T22:59:17.013 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-08T22:59:17.026 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../92-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-08T22:59:17.071 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T22:59:17.572 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pytest. 2022-03-08T22:59:17.585 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../93-python3-pytest_4.6.9-1_all.deb ... 2022-03-08T22:59:17.672 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-08T22:59:18.078 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplejson. 2022-03-08T22:59:18.091 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../94-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-08T22:59:18.135 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T22:59:18.588 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package radosgw. 2022-03-08T22:59:18.602 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../95-radosgw_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:18.654 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking radosgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:20.258 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package rbd-fuse. 2022-03-08T22:59:20.271 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../96-rbd-fuse_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:20.315 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking rbd-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:20.696 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-08T22:59:21.030 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T22:59:21.153 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T22:59:21.447 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cephadm (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:22.209 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user cephadm....done 2022-03-08T22:59:22.436 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-08T22:59:22.853 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-08T22:59:23.182 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-08T22:59:23.383 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T22:59:23.819 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-08T22:59:24.320 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T22:59:24.630 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-08T22:59:24.939 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T22:59:25.256 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T22:59:25.566 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-argparse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:25.891 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T22:59:26.017 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-08T22:59:26.302 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T22:59:26.662 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T22:59:27.061 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T22:59:27.422 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T22:59:27.908 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T22:59:28.232 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-08T22:59:28.583 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-08T22:59:29.000 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T22:59:29.151 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T22:59:29.445 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T22:59:29.947 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T22:59:30.363 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T22:59:30.765 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:31.076 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-08T22:59:31.478 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T22:59:31.711 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-08T22:59:32.044 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T22:59:32.170 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T22:59:32.305 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T22:59:32.728 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T22:59:33.042 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-08T22:59:33.382 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T22:59:33.534 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-08T22:59:33.898 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-08T22:59:34.235 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T22:59:34.570 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T22:59:34.659 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-08T22:59:34.906 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-08T22:59:35.046 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T22:59:35.406 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-08T22:59:35.790 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T22:59:35.933 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T22:59:36.102 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-08T22:59:36.467 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T22:59:36.777 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librados2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:36.920 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-08T22:59:38.701 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librgw2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:38.902 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:39.044 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:59:39.263 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-08T22:59:39.687 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-08T22:59:40.034 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-08T22:59:40.461 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-08T22:59:40.843 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:40.970 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T22:59:41.328 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libradosstriper1 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:41.522 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T22:59:41.941 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T22:59:43.137 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librbd1 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:43.311 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T22:59:43.462 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-08T22:59:43.622 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T22:59:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:59:44.090 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-08T22:59:44.441 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-08T22:59:44.750 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-modules-core (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:45.069 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:46.161 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T22:59:46.161 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T22:59:47.231 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs-dev (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:47.363 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T22:59:47.708 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-08T22:59:48.018 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T22:59:48.400 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rados (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:48.563 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rbd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:48.714 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up rbd-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:48.905 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-08T22:59:49.199 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-08T22:59:49.532 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:49.691 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cephfs (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:49.876 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T22:59:50.996 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:51.748 INFO:teuthology.orchestra.run.smithi186.stdout:Adding group ceph....done 2022-03-08T22:59:52.312 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user ceph....done 2022-03-08T22:59:52.556 INFO:teuthology.orchestra.run.smithi186.stdout:Setting system user ceph properties....done 2022-03-08T22:59:52.581 INFO:teuthology.orchestra.run.smithi186.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-08T22:59:52.722 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-08T22:59:53.297 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-08T22:59:54.260 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-test (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:54.487 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up radosgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:54.831 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T22:59:54.832 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T22:59:56.443 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-base (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:57.097 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-08T22:59:58.093 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mds (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:58.344 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T22:59:58.344 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T22:59:59.317 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:59.619 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T22:59:59.693 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T23:00:00.581 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-osd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:01.355 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T23:00:01.355 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T23:00:02.319 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:02.454 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:02.600 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mon (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:02.824 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T23:00:02.824 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T23:00:03.805 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:03.981 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-dashboard (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:04.126 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-volume (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T23:00:04.362 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-08T23:00:04.947 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T23:00:07.761 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T23:00:08.256 INFO:teuthology.orchestra.run.smithi186.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-08T23:00:08.259 DEBUG:teuthology.parallel:result is None 2022-03-08T23:00:08.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-08T23:00:08.308 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-08T23:00:08.313 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T23:00:08.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T23:00:08.470 DEBUG:teuthology.orchestra.run.smithi186:> dpkg-query -W -f '${Version}' ceph 2022-03-08T23:00:08.492 INFO:teuthology.orchestra.run.smithi186.stdout:17.0.0-11002-g94cb3430-1focal 2022-03-08T23:00:08.493 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11002-g94cb3430-1focal 2022-03-08T23:00:08.493 INFO:teuthology.task.install:The correct ceph version 17.0.0-11002-g94cb3430-1focal is installed. 2022-03-08T23:00:08.494 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-08T23:00:08.494 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T23:00:08.495 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-08T23:00:08.508 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-08T23:00:08.509 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T23:00:08.509 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2022-03-08T23:00:08.567 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-08T23:00:08.579 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-08T23:00:08.580 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-08T23:00:08.581 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-08T23:00:08.592 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-08T23:00:08.656 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-08T23:00:08.670 INFO:tasks.radosbench:Beginning radosbench... 2022-03-08T23:00:08.671 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-08T23:00:08.673 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd pool create unique_pool_0 16 2022-03-08T23:00:09.911 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'unique_pool_0' created 2022-03-08T23:00:09.932 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-08T23:00:11.127 INFO:teuthology.orchestra.run.smithi186.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-08T23:00:12.140 DEBUG:teuthology.orchestra.run.smithi186:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-08T23:00:12.143 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:00:12.153 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch host label add '`hostname`' 'foo 2022-03-08T23:00:12.154 DEBUG:teuthology.orchestra.run.smithi186:> ceph' orch host ls '|' grep 'foo 2022-03-08T23:00:12.154 DEBUG:teuthology.orchestra.run.smithi186:> ceph' orch host label rm '`hostname`' 'foo 2022-03-08T23:00:12.154 DEBUG:teuthology.orchestra.run.smithi186:> ceph' orch host ls '|' grep -v 'foo 2022-03-08T23:00:12.155 DEBUG:teuthology.orchestra.run.smithi186:> ' 2022-03-08T23:00:12.278 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:hints = 1 2022-03-08T23:00:12.278 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-08T23:00:12.278 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Object prefix: benchmark_data_smithi186_49573 2022-03-08T23:00:12.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:00:12.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 0 0 0 0 0 0 - 0 2022-03-08T23:00:12.518 INFO:teuthology.orchestra.run.smithi186.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-08T23:00:12.542 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T23:00:12.553 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T23:00:12.554 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi186.front.sepia.ceph.com 2022-03-08T23:00:12.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T23:00:12.554 DEBUG:teuthology.orchestra.run.smithi186:> toolbox() { 2022-03-08T23:00:12.555 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T23:00:12.555 DEBUG:teuthology.orchestra.run.smithi186:> } 2022-03-08T23:00:12.556 DEBUG:teuthology.orchestra.run.smithi186:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:00:12.556 DEBUG:teuthology.orchestra.run.smithi186:> toolbox ceph orch osd rm 0 --force 2022-03-08T23:00:12.556 DEBUG:teuthology.orchestra.run.smithi186:> removed_pv="" 2022-03-08T23:00:12.557 DEBUG:teuthology.orchestra.run.smithi186:> while [ "$removed_pv" = "" ] 2022-03-08T23:00:12.557 DEBUG:teuthology.orchestra.run.smithi186:> do 2022-03-08T23:00:12.557 DEBUG:teuthology.orchestra.run.smithi186:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T23:00:12.558 DEBUG:teuthology.orchestra.run.smithi186:> sleep 3s 2022-03-08T23:00:12.558 DEBUG:teuthology.orchestra.run.smithi186:> done 2022-03-08T23:00:12.558 DEBUG:teuthology.orchestra.run.smithi186:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T23:00:12.559 DEBUG:teuthology.orchestra.run.smithi186:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T23:00:12.559 DEBUG:teuthology.orchestra.run.smithi186:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T23:00:12.559 DEBUG:teuthology.orchestra.run.smithi186:> zap_completion="0" 2022-03-08T23:00:12.560 DEBUG:teuthology.orchestra.run.smithi186:> while [ "$zap_completion" = "0" ] 2022-03-08T23:00:12.560 DEBUG:teuthology.orchestra.run.smithi186:> do 2022-03-08T23:00:12.560 DEBUG:teuthology.orchestra.run.smithi186:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T23:00:12.560 DEBUG:teuthology.orchestra.run.smithi186:> sleep 3s 2022-03-08T23:00:12.561 DEBUG:teuthology.orchestra.run.smithi186:> done 2022-03-08T23:00:12.561 DEBUG:teuthology.orchestra.run.smithi186:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T23:00:12.561 DEBUG:teuthology.orchestra.run.smithi186:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:00:12.561 DEBUG:teuthology.orchestra.run.smithi186:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:00:12.562 DEBUG:teuthology.orchestra.run.smithi186:> num_osd="0" 2022-03-08T23:00:12.562 DEBUG:teuthology.orchestra.run.smithi186:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T23:00:12.562 DEBUG:teuthology.orchestra.run.smithi186:> do 2022-03-08T23:00:12.563 DEBUG:teuthology.orchestra.run.smithi186:> echo "waiting for osd to come back up" 2022-03-08T23:00:12.563 DEBUG:teuthology.orchestra.run.smithi186:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:00:12.563 DEBUG:teuthology.orchestra.run.smithi186:> sleep 30s 2022-03-08T23:00:12.563 DEBUG:teuthology.orchestra.run.smithi186:> done 2022-03-08T23:00:12.564 DEBUG:teuthology.orchestra.run.smithi186:> ' 2022-03-08T23:00:12.576 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:00:12.577 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:00:12.577 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:00:13.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 1 16 1272 1256 78.4957 78.5 0.0126027 0.0126838 2022-03-08T23:00:14.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 2 16 2798 2782 86.9311 95.375 0.0151416 0.0114679 2022-03-08T23:00:15.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 3 16 4252 4236 88.2427 90.875 0.0165574 0.0112992 2022-03-08T23:00:15.391 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:16.157 INFO:teuthology.orchestra.run.smithi186.stderr:+ orig_num_osd=4 2022-03-08T23:00:16.157 INFO:teuthology.orchestra.run.smithi186.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T23:00:16.157 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T23:00:16.263 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:16.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 4 16 5491 5475 85.5395 77.4375 0.0121782 0.0116675 2022-03-08T23:00:17.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 5 16 6555 6539 81.7305 66.5 0.0233547 0.012216 2022-03-08T23:00:18.194 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:18.193433 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T23:00:18.194 INFO:tasks.rook.operator.smithi186.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T23:00:18.195 INFO:tasks.rook.operator.smithi186.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T23:00:18.195 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-08T23:00:18.195 INFO:tasks.rook.operator.smithi186.stdout: Config: nil, 2022-03-08T23:00:18.195 INFO:tasks.rook.operator.smithi186.stdout: Selection: {}, 2022-03-08T23:00:18.196 INFO:tasks.rook.operator.smithi186.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:00:18.196 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:18.196 INFO:tasks.rook.operator.smithi186.stdout:- { 2022-03-08T23:00:18.196 INFO:tasks.rook.operator.smithi186.stdout:- Name: "smithi186-lv-2", 2022-03-08T23:00:18.197 INFO:tasks.rook.operator.smithi186.stdout:- Count: 1, 2022-03-08T23:00:18.197 INFO:tasks.rook.operator.smithi186.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:00:18.197 INFO:tasks.rook.operator.smithi186.stdout:- }, 2022-03-08T23:00:18.197 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:18.198 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:18.198 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-08T23:00:18.198 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-08T23:00:18.198 INFO:tasks.rook.operator.smithi186.stdout: Annotations: nil, 2022-03-08T23:00:18.199 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-08T23:00:18.199 INFO:tasks.rook.operator.smithi186.stdout: ... // 20 identical fields 2022-03-08T23:00:18.199 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-08T23:00:18.199 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:18.194031 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:18.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 6 16 7652 7636 79.5347 68.5625 0.0209659 0.0125323 2022-03-08T23:00:19.279 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 7 16 8463 8447 75.4129 50.6875 0.0159155 0.0130929 2022-03-08T23:00:20.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:20.085974 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:00:20.281 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 8 16 9404 9388 73.3245 58.8125 0.0297938 0.0136152 2022-03-08T23:00:21.281 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 9 16 10130 10114 70.2191 45.375 0.00712692 0.0139026 2022-03-08T23:00:21.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:21.563049 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:00:22.281 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 10 16 10130 10114 63.1982 0 - 0.0139026 2022-03-08T23:00:23.281 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 11 16 10130 10114 57.4535 0 - 0.0139026 2022-03-08T23:00:24.281 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 12 16 10130 10114 52.6663 0 - 0.0139026 2022-03-08T23:00:25.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 13 16 10130 10114 48.6154 0 - 0.0139026 2022-03-08T23:00:26.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 14 16 10130 10114 45.1431 0 - 0.0139026 2022-03-08T23:00:27.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 15 16 10130 10114 42.1339 0 - 0.0139026 2022-03-08T23:00:27.362 INFO:teuthology.orchestra.run.smithi186.stdout:Success 2022-03-08T23:00:27.401 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-08T23:00:27.401 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-08T23:00:27.402 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-08T23:00:27.403 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-08T23:00:27.403 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-08T23:00:27.704 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-08T23:00:27.705 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-08T23:00:28.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 16 16 10130 10114 39.5007 0 - 0.0139026 2022-03-08T23:00:29.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 17 16 10178 10162 37.3538 0.375 0.00243341 0.0220727 2022-03-08T23:00:30.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 18 16 10460 10444 36.2577 17.625 0.0144493 0.0275537 2022-03-08T23:00:30.706 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-08T23:00:30.707 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-08T23:00:30.707 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-08T23:00:30.708 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-08T23:00:30.816 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-08T23:00:30.816 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-08T23:00:31.282 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 19 16 11114 11098 36.5004 40.875 0.0117267 0.02712 2022-03-08T23:00:32.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:00:32.283927+0000 min lat: 0.00169504 max lat: 8.88076 avg lat: 0.026851 2022-03-08T23:00:32.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:00:32.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 20 16 11921 11905 37.1969 50.4375 0.0170864 0.026851 2022-03-08T23:00:33.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 21 16 12381 12365 36.7945 28.75 0.00853235 0.0270769 2022-03-08T23:00:33.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:33.298878 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T23:00:33.817 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-08T23:00:33.817 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-08T23:00:33.818 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-08T23:00:33.818 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-08T23:00:34.091 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv=smithi186-lv-2 2022-03-08T23:00:34.091 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-08T23:00:34.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 22 16 13169 13153 37.3604 49.25 0.0171443 0.0267484 2022-03-08T23:00:35.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 23 16 13753 13737 37.3229 36.5 0.0109011 0.0267798 2022-03-08T23:00:36.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 24 16 14610 14594 37.9993 53.5625 0.0223663 0.0263025 2022-03-08T23:00:37.092 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' smithi186-lv-2 = '' ']' 2022-03-08T23:00:37.092 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv smithi186-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-08T23:00:37.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 25 16 15138 15122 37.7992 33 0.0218691 0.0264379 2022-03-08T23:00:37.295 INFO:teuthology.orchestra.run.smithi186.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-08T23:00:37.296 INFO:teuthology.orchestra.run.smithi186.stderr:++ echo smithi186-lv-2 2022-03-08T23:00:37.296 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d - 2022-03-08T23:00:37.297 INFO:teuthology.orchestra.run.smithi186.stderr:+ host=smithi186 2022-03-08T23:00:37.297 INFO:teuthology.orchestra.run.smithi186.stderr:+ toolbox ceph orch device zap smithi186 /dev/vg_nvme/lv_2 --force 2022-03-08T23:00:37.298 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi186 /dev/vg_nvme/lv_2 --force 2022-03-08T23:00:37.524 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:38.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 26 16 15739 15723 37.7899 37.5625 0.0108134 0.0264445 2022-03-08T23:00:38.956 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_2 on smithi186 zapped 2022-03-08T23:00:38.970 INFO:teuthology.orchestra.run.smithi186.stderr:+ zap_completion=0 2022-03-08T23:00:38.971 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 0 = 0 ']' 2022-03-08T23:00:38.971 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T23:00:39.283 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 27 16 16153 16137 37.3485 25.875 0.0203942 0.0267602 2022-03-08T23:00:39.332 INFO:teuthology.orchestra.run.smithi186.stderr:+ zap_completion= 2022-03-08T23:00:39.332 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-08T23:00:40.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 28 16 16859 16843 37.5903 44.125 0.0204239 0.0265655 2022-03-08T23:00:41.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 29 16 17599 17583 37.8887 46.25 0.068417 0.0263409 2022-03-08T23:00:42.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 30 16 18379 18363 38.2506 48.75 0.0175337 0.026128 2022-03-08T23:00:42.336 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = 0 ']' 2022-03-08T23:00:42.337 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl patch pv smithi186-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-08T23:00:43.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 31 16 19120 19104 38.5105 46.3125 0.015399 0.0259556 2022-03-08T23:00:43.463 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-2 patched 2022-03-08T23:00:43.465 INFO:teuthology.orchestra.run.smithi186.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:00:43.465 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T23:00:43.739 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:44.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 32 16 19840 19824 38.7131 45 0.0196741 0.0258147 2022-03-08T23:00:45.129 INFO:teuthology.orchestra.run.smithi186.stdout:['Success'] 2022-03-08T23:00:45.132 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:45.130916 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T23:00:45.132 INFO:tasks.rook.operator.smithi186.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T23:00:45.132 INFO:tasks.rook.operator.smithi186.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T23:00:45.132 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-08T23:00:45.133 INFO:tasks.rook.operator.smithi186.stdout: Config: nil, 2022-03-08T23:00:45.133 INFO:tasks.rook.operator.smithi186.stdout: Selection: {}, 2022-03-08T23:00:45.133 INFO:tasks.rook.operator.smithi186.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:00:45.133 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:45.134 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:45.134 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:45.134 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-08T23:00:45.134 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-2", 2022-03-08T23:00:45.135 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-08T23:00:45.135 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:00:45.135 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-08T23:00:45.135 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-08T23:00:45.135 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-08T23:00:45.136 INFO:tasks.rook.operator.smithi186.stdout: Annotations: nil, 2022-03-08T23:00:45.136 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-08T23:00:45.136 INFO:tasks.rook.operator.smithi186.stdout: ... // 20 identical fields 2022-03-08T23:00:45.136 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-08T23:00:45.147 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:00:45.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 33 16 20509 20493 38.8069 41.8125 0.0256409 0.0257577 2022-03-08T23:00:45.751 INFO:teuthology.orchestra.run.smithi186.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T23:00:46.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 34 16 20951 20935 38.478 27.625 0.016001 0.0259755 2022-03-08T23:00:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=0 2022-03-08T23:00:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 0 '!=' 4 ']' 2022-03-08T23:00:46.632 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:00:46.633 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:00:46.633 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:00:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:00:46.634 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:00:46.987 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:47.284 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 35 16 21589 21573 38.5178 39.875 0.00987813 0.0259502 2022-03-08T23:00:48.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:48.124771 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:00:48.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:48.124794 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:00:48.127 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:48.124802 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:00:48.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 36 16 22227 22211 38.5554 39.875 0.0188668 0.0259216 2022-03-08T23:00:48.473 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-08T23:00:48.473 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:00:48.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:48.791701 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:00:49.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 37 16 22933 22917 38.7058 44.125 0.017836 0.0258243 2022-03-08T23:00:49.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:49.403183 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:00:49.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:49.488625 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:00:50.218 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:50.218458 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:00:50.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 38 16 23496 23480 38.6131 35.1875 0.010681 0.0258849 2022-03-08T23:00:51.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 39 16 24169 24153 38.7015 42.0625 0.0218417 0.0258291 2022-03-08T23:00:51.510 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:51.510558 I | op-mon: start running mons 2022-03-08T23:00:51.531 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:51.530130 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:00:51.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:51.713948 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T23:00:51.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:51.739626 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:00:51.744 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:51.739959 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:00:52.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:00:52.285985+0000 min lat: 0.00169504 max lat: 8.88076 avg lat: 0.0257251 2022-03-08T23:00:52.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:00:52.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 40 16 24886 24870 38.8541 44.8125 0.024371 0.0257251 2022-03-08T23:00:52.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:52.327975 I | op-mon: targeting the mon count 1 2022-03-08T23:00:52.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:52.403177 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T23:00:52.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:52.403265 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:53.048 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:53.048426 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:00:53.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:53.064621 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:00:53.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 41 16 25316 25300 38.5619 26.875 0.0155441 0.0258978 2022-03-08T23:00:53.616 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:53.616041 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T23:00:54.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 42 16 25810 25794 38.3788 30.875 0.0210319 0.0260423 2022-03-08T23:00:54.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:54.493307 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:00:55.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 43 16 26426 26410 38.3815 38.5 0.0179143 0.0260437 2022-03-08T23:00:56.285 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 44 16 27143 27127 38.5276 44.8125 0.0182971 0.0259445 2022-03-08T23:00:56.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:56.595237 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:00:57.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 45 16 27605 27589 38.313 28.875 0.023133 0.0260876 2022-03-08T23:00:58.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 46 16 28303 28287 38.4284 43.625 0.0198008 0.0260037 2022-03-08T23:00:58.761 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:00:58.760932 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:00:59.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 47 16 28575 28559 37.9725 17 0.0192057 0.0263238 2022-03-08T23:01:00.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 48 16 29200 29184 37.9951 39.0625 0.0250196 0.0263094 2022-03-08T23:01:01.024 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:00.989038 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:01:01.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 49 16 29823 29807 38.0142 38.9375 0.0126092 0.0262964 2022-03-08T23:01:02.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 50 16 30553 30537 38.1664 45.625 0.0186166 0.0261892 2022-03-08T23:01:02.999 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:02.998830 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:01:03.286 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 51 16 31050 31034 38.027 31.0625 0.0159688 0.026286 2022-03-08T23:01:04.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 52 16 31778 31762 38.1706 45.5 0.00742376 0.0261857 2022-03-08T23:01:05.047 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:05.046663 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T23:01:05.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 53 16 32449 32433 38.2416 41.9375 0.0210275 0.0261405 2022-03-08T23:01:06.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 54 16 33138 33122 38.3308 43.0625 0.0144994 0.0260785 2022-03-08T23:01:07.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 55 16 33686 33670 38.2566 34.25 0.0274212 0.0260596 2022-03-08T23:01:08.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 56 16 34046 34030 37.9752 22.5 0.0178995 0.0263214 2022-03-08T23:01:09.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 57 16 34341 34325 37.6324 18.4375 0.128652 0.0263958 2022-03-08T23:01:10.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 58 16 34341 34325 36.9836 0 - 0.0263958 2022-03-08T23:01:11.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 59 15 34686 34671 36.7232 10.8125 0.0238682 0.0272179 2022-03-08T23:01:12.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:01:12.288005+0000 min lat: 0.00169504 max lat: 8.88076 avg lat: 0.0271999 2022-03-08T23:01:12.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:01:12.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 60 16 35302 35286 36.7517 38.4375 0.0216812 0.0271999 2022-03-08T23:01:13.287 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 61 16 35788 35772 36.6471 30.375 0.0224969 0.0272062 2022-03-08T23:01:14.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 62 16 36041 36025 36.3111 15.8125 0.102464 0.0275276 2022-03-08T23:01:15.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 63 16 36682 36666 36.3705 40.0625 0.0191388 0.0274831 2022-03-08T23:01:16.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 64 16 37369 37353 36.4731 42.9375 0.0303921 0.0274056 2022-03-08T23:01:17.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 65 16 37874 37858 36.3974 31.5625 0.0390335 0.0274651 2022-03-08T23:01:18.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 66 16 38368 38352 36.3136 30.875 0.0201466 0.0275273 2022-03-08T23:01:18.474 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:01:18.475 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:01:18.476 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:01:18.476 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:01:18.476 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:01:18.476 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:01:19.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 67 16 38848 38832 36.2194 30 0.0132833 0.0276005 2022-03-08T23:01:19.467 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:20.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 68 16 38938 38922 35.7695 5.625 0.0202792 0.0279262 2022-03-08T23:01:21.288 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 69 16 39406 39390 35.6749 29.25 0.0241442 0.0280176 2022-03-08T23:01:21.459 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-08T23:01:21.460 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:01:22.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 70 16 40006 39990 35.701 37.5 0.0138207 0.0280041 2022-03-08T23:01:23.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 71 16 40612 40596 35.7315 37.875 0.0583779 0.0279605 2022-03-08T23:01:24.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 72 16 41023 41007 35.592 25.6875 0.244447 0.0280821 2022-03-08T23:01:25.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 73 16 41528 41512 35.5367 31.5625 0.0188641 0.0281342 2022-03-08T23:01:26.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 74 16 42164 42148 35.5936 39.75 0.0221304 0.0280824 2022-03-08T23:01:27.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 75 16 42674 42658 35.544 31.875 0.0188875 0.0281241 2022-03-08T23:01:28.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 76 16 43219 43203 35.5245 34.0625 0.0177688 0.0281387 2022-03-08T23:01:29.289 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 77 16 43664 43648 35.4243 27.8125 0.0212554 0.0281981 2022-03-08T23:01:30.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 78 16 43932 43916 35.1849 16.75 0.0178172 0.0283759 2022-03-08T23:01:31.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 79 16 44075 44059 34.8526 8.9375 0.00595317 0.0284437 2022-03-08T23:01:32.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:01:32.290202+0000 min lat: 0.00169504 max lat: 8.88076 avg lat: 0.0284437 2022-03-08T23:01:32.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:01:32.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 80 16 44075 44059 34.417 0 - 0.0284437 2022-03-08T23:01:33.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 81 16 44075 44059 33.9921 0 - 0.0284437 2022-03-08T23:01:34.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 82 16 44075 44059 33.5775 0 - 0.0284437 2022-03-08T23:01:35.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 83 16 44075 44059 33.173 0 - 0.0284437 2022-03-08T23:01:36.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 84 16 44075 44059 32.7781 0 - 0.0284437 2022-03-08T23:01:37.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 85 16 44075 44059 32.3925 0 - 0.0284437 2022-03-08T23:01:38.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 86 16 44075 44059 32.0158 0 - 0.0284437 2022-03-08T23:01:39.290 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 87 16 44075 44059 31.6478 0 - 0.0284437 2022-03-08T23:01:40.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 88 16 44075 44059 31.2882 0 - 0.0284437 2022-03-08T23:01:41.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 89 16 44266 44250 31.0707 1.19375 0.00423048 0.0320414 2022-03-08T23:01:42.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 90 16 44266 44250 30.7255 0 - 0.0320414 2022-03-08T23:01:43.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 91 16 44266 44250 30.3879 0 - 0.0320414 2022-03-08T23:01:44.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 92 16 44266 44250 30.0576 0 - 0.0320414 2022-03-08T23:01:45.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 93 16 44266 44250 29.7344 0 - 0.0320414 2022-03-08T23:01:46.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 94 16 44266 44250 29.4181 0 - 0.0320414 2022-03-08T23:01:47.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 95 16 44266 44250 29.1084 0 - 0.0320414 2022-03-08T23:01:48.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 96 16 44266 44250 28.8052 0 - 0.0320414 2022-03-08T23:01:49.291 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 97 16 44641 44625 28.7499 2.92969 0.0139301 0.0347731 2022-03-08T23:01:50.292 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 98 16 45208 45192 28.8181 35.4375 0.0209205 0.0346909 2022-03-08T23:01:50.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:50.421973 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:01:50.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:50.422098 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:01:50.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:50.422121 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:01:50.641 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:50.640380 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:01:50.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:50.782985 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:50.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:50.783219 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:51.292 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 99 16 45718 45702 28.8489 31.875 0.0220044 0.0346539 2022-03-08T23:01:51.461 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:01:51.461 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:01:51.462 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:01:51.462 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:01:51.462 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:01:51.462 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:01:51.565 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:51.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:51.604235 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:01:52.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:52.030692 I | op-mon: start running mons 2022-03-08T23:01:52.292 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:01:52.292230+0000 min lat: 0.00169504 max lat: 10.6739 avg lat: 0.0346963 2022-03-08T23:01:52.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:01:52.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 100 16 46124 46108 28.8141 25.375 0.0215587 0.0346963 2022-03-08T23:01:52.351 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-08T23:01:53.293 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:01:53.294 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:52.478501 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:01:53.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 101 16 46565 46549 28.8017 27.5625 0.204287 0.0347067 2022-03-08T23:01:53.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:53.754905 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T23:01:54.290 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:54.170094 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:54.293 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 102 16 47052 47036 28.8173 30.4375 0.113069 0.0346678 2022-03-08T23:01:54.294 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:54.290418 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:54.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:54.328324 I | op-mon: targeting the mon count 1 2022-03-08T23:01:54.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:54.336960 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:55.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 103 16 47626 47610 28.8857 35.875 0.0119648 0.0346102 2022-03-08T23:01:55.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:55.376944 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:55.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:55.376972 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:56.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:56.015540 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:56.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:56.015565 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:56.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 104 16 48178 48162 28.9396 34.5 0.0319888 0.0345438 2022-03-08T23:01:56.880 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:56.878930 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:56.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:56.878957 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:57.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 105 16 48672 48656 28.9581 30.875 0.0351776 0.0345238 2022-03-08T23:01:57.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:57.606901 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:57.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:57.606930 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:01:58.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:58.183922 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:01:58.185 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:58.183942 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:01:58.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 106 16 49173 49157 28.9802 31.3125 0.0190642 0.0344952 2022-03-08T23:01:58.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:58.885122 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:01:58.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:58.885141 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:01:59.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.014922 I | op-mon: mon "a" endpoint is [v2:10.107.60.203:3300,v1:10.107.60.203:6789] 2022-03-08T23:01:59.097 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.097319 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T23:01:59.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.108035 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:59.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.108221 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:59.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 107 16 49729 49713 29.0341 34.75 0.0238665 0.0344324 2022-03-08T23:01:59.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.304809 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:01:59.323 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.321208 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:01:59.323 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.321237 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:01:59.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:01:59.495750 I | op-mon: mons running: [a] 2022-03-08T23:02:00.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 108 16 50276 50260 29.0818 34.1875 0.040267 0.0343783 2022-03-08T23:02:00.370 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:00.367929 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:00.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:00.367956 I | op-mon: mons created: 1 2022-03-08T23:02:01.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:01.165632 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:01.186 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:01.173370 I | op-mon: mons running: [a] 2022-03-08T23:02:01.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 109 16 50818 50802 29.1257 33.875 0.0251524 0.0343251 2022-03-08T23:02:01.990 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:01.989214 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:01.992 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:01.989248 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:02:02.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 110 16 51370 51354 29.1746 34.5 0.0268612 0.0342659 2022-03-08T23:02:02.791 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:02.790752 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:02:03.294 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 111 16 51880 51864 29.1989 31.875 0.0378932 0.034236 2022-03-08T23:02:03.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:03.654894 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:02:04.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 112 16 52396 52380 29.2256 32.25 0.0215887 0.0342061 2022-03-08T23:02:04.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:04.360770 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:02:05.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 113 16 52830 52814 29.207 27.125 0.0342219 0.0341671 2022-03-08T23:02:05.826 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:05.824819 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:02:05.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:05.824846 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:02:06.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 114 16 53207 53191 29.1575 23.5625 0.0324404 0.0342772 2022-03-08T23:02:06.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:06.439582 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:02:07.001 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:07.001447 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:02:07.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:07.002125 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:02:07.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:07.002142 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:02:07.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 115 16 53484 53468 29.0545 17.3125 0.0299099 0.0344101 2022-03-08T23:02:07.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:07.960925 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:02:08.023 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:08.019727 I | op-mgr: start running mgr 2022-03-08T23:02:08.023 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:08.019756 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:02:08.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 116 16 53959 53943 29.0599 29.6875 0.0239017 0.0344017 2022-03-08T23:02:09.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.252578 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:02:09.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.268152 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:02:09.273 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.270966 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:02:09.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 117 16 54278 54262 28.9819 19.9375 0.0695936 0.0344961 2022-03-08T23:02:09.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.444274 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:02:09.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.444299 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:02:09.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.477233 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:02:09.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.523947 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-1" 2022-03-08T23:02:09.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.524021 I | op-osd: OSD PVC "smithi186-lv-1-data-0b8dfr" already exists 2022-03-08T23:02:09.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.524051 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-3" 2022-03-08T23:02:09.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.524092 I | op-osd: OSD PVC "smithi186-lv-3-data-04q5qg" already exists 2022-03-08T23:02:09.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.524122 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-4" 2022-03-08T23:02:09.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.524146 I | op-osd: OSD PVC "smithi186-lv-4-data-0r9vqb" already exists 2022-03-08T23:02:09.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.524186 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-2" 2022-03-08T23:02:09.536 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.532407 I | op-osd: successfully provisioned PVC "smithi186-lv-2-data-078mw5" 2022-03-08T23:02:09.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.564596 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0b8dfr" 2022-03-08T23:02:09.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.564624 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-04q5qg" 2022-03-08T23:02:09.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.564630 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0r9vqb" 2022-03-08T23:02:09.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.564635 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-078mw5" 2022-03-08T23:02:09.602 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.599927 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-2-data-078mw5" 2022-03-08T23:02:09.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.599959 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:02:09.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.599966 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:02:09.616 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:09.615455 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-078mw5 is "starting" 2022-03-08T23:02:10.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 118 16 54560 54544 28.8856 17.625 0.146054 0.034598 2022-03-08T23:02:11.297 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 119 16 54996 54980 28.8719 27.25 0.0144472 0.0346251 2022-03-08T23:02:12.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:12.013961 I | op-mgr: successful modules: dashboard 2022-03-08T23:02:12.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:12.027768 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:02:12.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:12.032712 I | op-mgr: successful modules: prometheus 2022-03-08T23:02:12.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:02:12.298181+0000 min lat: 0.00169504 max lat: 10.6739 avg lat: 0.0346207 2022-03-08T23:02:12.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:02:12.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 120 16 55470 55454 28.8781 29.625 0.0357659 0.0346207 2022-03-08T23:02:12.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:12.770900 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:02:12.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:12.780410 I | op-osd: updating OSD 1 on PVC "smithi186-lv-3-data-04q5qg" 2022-03-08T23:02:12.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:12.780436 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-04q5qg" 2022-03-08T23:02:13.109 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:13.108924 I | op-mgr: successful modules: balancer 2022-03-08T23:02:13.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 121 16 55912 55896 28.8677 27.625 0.0339548 0.0346287 2022-03-08T23:02:14.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 122 16 56371 56355 28.8662 28.6875 0.0154975 0.0346334 2022-03-08T23:02:15.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 123 16 56886 56870 28.8932 32.1875 0.0194064 0.0346019 2022-03-08T23:02:16.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 124 16 57347 57331 28.8925 28.8125 0.223982 0.034585 2022-03-08T23:02:16.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:16.373814 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:02:17.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:17.167506 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:02:17.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 125 16 57823 57807 28.8993 29.75 0.0175818 0.0345934 2022-03-08T23:02:17.373 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:17.372472 I | op-osd: updating OSD 2 on PVC "smithi186-lv-1-data-0b8dfr" 2022-03-08T23:02:17.373 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:17.372498 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0b8dfr" 2022-03-08T23:02:18.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 126 16 58283 58267 28.8981 28.75 0.0364718 0.0345848 2022-03-08T23:02:19.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 127 16 58759 58743 28.9048 29.75 0.0146554 0.0345852 2022-03-08T23:02:19.976 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:19.973047 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:02:20.298 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 128 15 59241 59226 28.9148 30.1875 0.0260345 0.0345761 2022-03-08T23:02:20.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:20.428624 I | op-osd: updating OSD 3 on PVC "smithi186-lv-4-data-0r9vqb" 2022-03-08T23:02:20.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:20.428737 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0r9vqb" 2022-03-08T23:02:21.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 129 15 59586 59571 28.8578 21.5625 0.0364526 0.0346419 2022-03-08T23:02:22.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 130 16 59978 59962 28.8238 24.4375 0.0271784 0.0346847 2022-03-08T23:02:22.352 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:02:22.353 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:02:22.353 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:02:22.353 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:02:22.353 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:02:22.354 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:02:22.560 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:02:23.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 131 16 60477 60461 28.8418 31.1875 0.0384997 0.0346631 2022-03-08T23:02:23.343 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-08T23:02:23.344 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:02:24.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 132 16 60997 60981 28.8695 32.5 0.0218409 0.0346277 2022-03-08T23:02:25.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 133 16 61427 61411 28.8545 26.875 0.0426198 0.0346485 2022-03-08T23:02:26.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 134 16 61853 61837 28.8378 26.625 0.0144515 0.0346677 2022-03-08T23:02:27.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 135 16 62296 62280 28.8293 27.6875 0.0238534 0.0346726 2022-03-08T23:02:28.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 136 16 62813 62797 28.8549 32.3125 0.0214041 0.0346473 2022-03-08T23:02:29.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 137 16 63121 63105 28.7847 19.25 0.0302126 0.0347315 2022-03-08T23:02:30.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 138 16 63599 63583 28.7926 29.875 0.0147935 0.0346992 2022-03-08T23:02:31.299 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 139 16 64146 64130 28.8314 34.1875 0.0368431 0.0346755 2022-03-08T23:02:32.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:02:32.300111+0000 min lat: 0.00169504 max lat: 10.6739 avg lat: 0.0346378 2022-03-08T23:02:32.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:02:32.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 140 16 64680 64664 28.8638 33.375 0.0359137 0.0346378 2022-03-08T23:02:33.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 141 16 65245 65229 28.9095 35.3125 0.0202348 0.0345813 2022-03-08T23:02:34.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 142 16 65566 65550 28.8472 20.0625 0.00713903 0.0345587 2022-03-08T23:02:35.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 143 16 65566 65550 28.6455 0 - 0.0345587 2022-03-08T23:02:36.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 144 16 65566 65550 28.4466 0 - 0.0345587 2022-03-08T23:02:36.515 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:36.515495 I | op-osd: OSD orchestration status for node smithi186-lv-2-data-078mw5 is "orchestrating" 2022-03-08T23:02:36.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:36.945610 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-078mw5 is "orchestrating" 2022-03-08T23:02:37.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 145 16 65566 65550 28.2504 0 - 0.0345587 2022-03-08T23:02:38.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 146 16 65566 65550 28.0569 0 - 0.0345587 2022-03-08T23:02:39.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 147 16 65566 65550 27.8661 0 - 0.0345587 2022-03-08T23:02:40.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 148 16 65566 65550 27.6778 0 - 0.0345587 2022-03-08T23:02:41.300 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 149 16 65566 65550 27.492 0 - 0.0345587 2022-03-08T23:02:42.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 150 16 65566 65550 27.3087 0 - 0.0345587 2022-03-08T23:02:43.253 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:43.253148 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-078mw5 is "completed" 2022-03-08T23:02:43.254 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:43.253176 I | op-osd: creating OSD 0 on PVC "smithi186-lv-2-data-078mw5" 2022-03-08T23:02:43.254 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:43.253185 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-078mw5" 2022-03-08T23:02:43.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 151 16 65566 65550 27.1279 0 - 0.0345587 2022-03-08T23:02:44.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 152 16 65566 65550 26.9494 0 - 0.0345587 2022-03-08T23:02:45.043 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:45.043409 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:02:45.044 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:45.043427 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:45.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:45.061409 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:45.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:45.190256 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:02:45.218 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:45.217001 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:02:45.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 153 16 65785 65769 26.8628 1.24432 0.00957964 0.0371821 2022-03-08T23:02:45.529 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:02:45.529313 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:02:45.473960601 +0000 UTC m=+546.928238118 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:02:46.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 154 16 65861 65845 26.7192 4.75 0.00322886 0.0371482 2022-03-08T23:02:47.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 155 16 65861 65845 26.5468 0 - 0.0371482 2022-03-08T23:02:48.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 156 16 65861 65845 26.3766 0 - 0.0371482 2022-03-08T23:02:49.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 157 16 65861 65845 26.2086 0 - 0.0371482 2022-03-08T23:02:50.301 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 158 16 65861 65845 26.0427 0 - 0.0371482 2022-03-08T23:02:51.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 159 16 65861 65845 25.879 0 - 0.0371482 2022-03-08T23:02:52.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:02:52.302217+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0371482 2022-03-08T23:02:52.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:02:52.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 160 16 65861 65845 25.7172 0 - 0.0371482 2022-03-08T23:02:53.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 161 16 66041 66025 25.6274 1.60714 0.0311605 0.0390124 2022-03-08T23:02:53.344 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:02:53.345 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:02:53.345 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:02:53.345 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:02:53.348 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:02:53.348 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:02:53.470 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:02:54.263 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-08T23:02:54.263 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:02:54.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 162 16 66439 66423 25.6227 24.875 0.0394048 0.0390183 2022-03-08T23:02:55.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 163 16 66839 66823 25.6189 25 0.0336109 0.039023 2022-03-08T23:02:56.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 164 16 67354 67338 25.6589 32.1875 0.0212373 0.0389639 2022-03-08T23:02:57.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 165 16 67778 67762 25.664 26.5 0.0188492 0.0389562 2022-03-08T23:02:58.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 166 16 68170 68154 25.657 24.5 0.0652637 0.0389638 2022-03-08T23:02:59.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 167 15 68573 68558 25.6545 25.25 0.0184939 0.0389717 2022-03-08T23:03:00.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 168 16 69018 69002 25.667 27.75 0.0257836 0.0389515 2022-03-08T23:03:01.302 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 169 16 69394 69378 25.6541 23.5 0.0204262 0.0389388 2022-03-08T23:03:02.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 170 16 69840 69824 25.6672 27.875 0.0138804 0.0389512 2022-03-08T23:03:03.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 171 16 70342 70326 25.7005 31.375 0.0204386 0.0389007 2022-03-08T23:03:04.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 172 16 70805 70789 25.7193 28.9375 0.0285224 0.0388696 2022-03-08T23:03:05.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 173 16 71213 71197 25.7181 25.5 0.0195172 0.0388745 2022-03-08T23:03:06.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 174 16 71630 71614 25.72 26.0625 0.0420292 0.0388548 2022-03-08T23:03:07.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 175 16 72057 72041 25.7255 26.6875 0.033883 0.0388632 2022-03-08T23:03:08.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 176 16 72476 72460 25.7281 26.1875 0.0334498 0.0388592 2022-03-08T23:03:09.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 177 16 72844 72828 25.7127 23 0.144286 0.0388765 2022-03-08T23:03:10.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:10.091080 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:03:10.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:10.091103 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:03:10.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:10.091113 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:03:10.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 178 16 73298 73282 25.7276 28.375 0.0289316 0.0388584 2022-03-08T23:03:10.393 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:10.393146 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:03:10.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:10.519483 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:03:10.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:10.519928 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:03:11.303 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 179 16 73646 73630 25.7054 21.75 0.0276043 0.0388928 2022-03-08T23:03:11.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:11.580193 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:03:12.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:12.175006 I | op-mon: start running mons 2022-03-08T23:03:12.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:03:12.304241+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0388613 2022-03-08T23:03:12.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:03:12.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 180 16 74118 74102 25.7265 29.5 0.0313514 0.0388613 2022-03-08T23:03:12.526 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:12.526492 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:03:13.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 181 16 74470 74454 25.7059 22 0.0330926 0.0388932 2022-03-08T23:03:13.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:13.997682 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T23:03:14.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 182 16 74839 74823 25.6913 23.0625 0.0392367 0.0389138 2022-03-08T23:03:14.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:14.325630 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:03:14.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:14.325894 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:03:14.459 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:14.459095 I | op-mon: targeting the mon count 1 2022-03-08T23:03:14.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:14.467135 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:03:15.102 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:15.102389 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:03:15.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:15.102412 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:03:15.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 183 16 75237 75221 25.6869 24.875 0.0235696 0.0389189 2022-03-08T23:03:15.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:15.897014 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:03:15.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:15.897037 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:03:16.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 184 16 75681 75665 25.6981 27.75 0.0171732 0.038901 2022-03-08T23:03:16.760 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:16.750913 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:03:16.761 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:16.750938 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:03:17.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:17.187680 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:03:17.206 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:17.187706 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:03:17.304 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 185 16 75978 75962 25.6595 18.5625 0.0172759 0.0389643 2022-03-08T23:03:18.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:18.135645 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:03:18.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:18.135670 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:03:18.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 186 16 76378 76362 25.6559 25 0.0422417 0.0389693 2022-03-08T23:03:18.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:18.684931 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:03:18.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:18.689005 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:03:19.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 187 16 76752 76736 25.6437 23.375 0.0226561 0.0389874 2022-03-08T23:03:19.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.849084 I | op-mon: mon "a" endpoint is [v2:10.107.60.203:3300,v1:10.107.60.203:6789] 2022-03-08T23:03:19.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.942238 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.60.203:6789"]}] data:a=10.107.60.203:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-08T23:03:19.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.955054 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:03:19.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.955231 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:03:19.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.977970 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:03:19.985 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.985875 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:03:19.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:19.985903 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:03:20.001 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:20.001101 I | op-mon: mons running: [a] 2022-03-08T23:03:20.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 188 16 77168 77152 25.6455 26 0.0758552 0.0389842 2022-03-08T23:03:20.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:20.718522 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:03:20.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:20.718546 I | op-mon: mons created: 1 2022-03-08T23:03:21.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 189 16 77585 77569 25.6477 26.0625 0.0228423 0.0389799 2022-03-08T23:03:21.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:21.430573 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:03:21.451 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:21.451287 I | op-mon: mons running: [a] 2022-03-08T23:03:22.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 190 16 77994 77978 25.6473 25.5625 0.0384162 0.038982 2022-03-08T23:03:22.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:22.329160 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:03:22.330 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:22.329195 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:03:23.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 191 16 78347 78331 25.6285 22.0625 0.024364 0.0390109 2022-03-08T23:03:23.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:23.879358 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:03:24.264 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:03:24.265 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:03:24.265 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:03:24.266 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:03:24.266 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:03:24.266 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:03:24.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 192 16 78779 78763 25.6356 27 0.0725065 0.0389985 2022-03-08T23:03:24.364 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:03:24.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:24.678121 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:03:25.176 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-08T23:03:25.176 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:03:25.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 193 16 79145 79129 25.6213 22.875 0.0479457 0.0390207 2022-03-08T23:03:25.459 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:25.459561 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:03:26.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:26.210675 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:03:26.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:26.210699 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:03:26.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 194 16 79563 79547 25.6239 26.125 0.0425426 0.0390174 2022-03-08T23:03:27.092 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:27.092255 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:03:27.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 195 16 79948 79932 25.6159 24.0625 0.0858327 0.0390275 2022-03-08T23:03:27.878 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:27.876605 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:03:27.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:27.876637 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:03:27.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:27.876643 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:03:28.305 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 196 16 80338 80322 25.6095 24.375 0.0873186 0.039037 2022-03-08T23:03:28.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:28.682866 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:03:28.730 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:28.730672 I | op-mgr: start running mgr 2022-03-08T23:03:28.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:28.730694 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:03:29.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.223751 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:03:29.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.231673 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:03:29.234 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.233658 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:03:29.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 197 16 80484 80468 25.5259 9.125 0.0458327 0.0390345 2022-03-08T23:03:29.363 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.362387 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:03:29.363 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.362412 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:03:29.389 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.389545 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:03:29.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393228 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-1" 2022-03-08T23:03:29.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393262 I | op-osd: OSD PVC "smithi186-lv-1-data-0b8dfr" already exists 2022-03-08T23:03:29.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393289 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-3" 2022-03-08T23:03:29.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393302 I | op-osd: OSD PVC "smithi186-lv-3-data-04q5qg" already exists 2022-03-08T23:03:29.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393310 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-4" 2022-03-08T23:03:29.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393353 I | op-osd: OSD PVC "smithi186-lv-4-data-0r9vqb" already exists 2022-03-08T23:03:29.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393363 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-2" 2022-03-08T23:03:29.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.393372 I | op-osd: OSD PVC "smithi186-lv-2-data-078mw5" already exists 2022-03-08T23:03:29.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.403427 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0b8dfr" 2022-03-08T23:03:29.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.403453 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-04q5qg" 2022-03-08T23:03:29.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.403459 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0r9vqb" 2022-03-08T23:03:29.408 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.403464 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-078mw5" 2022-03-08T23:03:29.408 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.403485 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:03:29.408 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:29.403489 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:03:30.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.223355 I | op-mgr: successful modules: prometheus 2022-03-08T23:03:30.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.229659 I | op-mgr: successful modules: dashboard 2022-03-08T23:03:30.233 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.230447 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:03:30.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 198 16 80484 80468 25.3969 0 - 0.0390345 2022-03-08T23:03:30.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.396932 I | op-mgr: successful modules: balancer 2022-03-08T23:03:30.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.663678 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:30.670 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.670089 I | op-osd: updating OSD 0 on PVC "smithi186-lv-2-data-078mw5" 2022-03-08T23:03:30.670 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:30.670106 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-078mw5" 2022-03-08T23:03:31.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 199 16 80484 80468 25.2693 0 - 0.0390345 2022-03-08T23:03:31.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:31.956771 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:03:32.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:03:32.306618+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0390345 2022-03-08T23:03:32.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:03:32.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 200 16 80484 80468 25.143 0 - 0.0390345 2022-03-08T23:03:32.925 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:32.925302 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:32.929 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:32.929775 I | op-osd: updating OSD 1 on PVC "smithi186-lv-3-data-04q5qg" 2022-03-08T23:03:32.929 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:32.929803 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-04q5qg" 2022-03-08T23:03:33.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 201 16 80594 80578 25.0521 1.71875 0.025849 0.0399048 2022-03-08T23:03:34.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 202 16 80880 80864 25.0166 17.875 0.028415 0.0399594 2022-03-08T23:03:35.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 203 16 81153 81137 24.9774 17.0625 0.0519712 0.0400242 2022-03-08T23:03:35.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:35.310971 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:35.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:35.318459 I | op-osd: updating OSD 2 on PVC "smithi186-lv-1-data-0b8dfr" 2022-03-08T23:03:35.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:35.318484 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0b8dfr" 2022-03-08T23:03:36.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 204 16 81415 81399 24.9352 16.375 0.0375065 0.0400915 2022-03-08T23:03:37.306 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 205 16 81667 81651 24.8904 15.75 0.0322239 0.0401661 2022-03-08T23:03:37.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:37.586844 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:37.592 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:37.591494 I | op-osd: updating OSD 3 on PVC "smithi186-lv-4-data-0r9vqb" 2022-03-08T23:03:37.592 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:37.591520 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0r9vqb" 2022-03-08T23:03:38.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 206 16 81915 81899 24.8448 15.5 0.0203096 0.0402371 2022-03-08T23:03:38.452 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:38.451661 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:03:38.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:03:38.451689 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:03:39.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 207 16 82211 82195 24.8141 18.5 0.019327 0.0402873 2022-03-08T23:03:40.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 208 16 82521 82505 24.788 19.375 0.0361915 0.040332 2022-03-08T23:03:41.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 209 16 82788 82772 24.7492 16.6875 0.049246 0.0403956 2022-03-08T23:03:42.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 210 16 83024 83008 24.7016 14.75 0.02148 0.0404344 2022-03-08T23:03:43.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 211 16 83195 83179 24.6352 10.6875 0.112148 0.0405801 2022-03-08T23:03:44.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 212 16 83506 83490 24.6106 19.4375 0.0325316 0.040617 2022-03-08T23:03:45.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 213 16 83804 83788 24.5825 18.625 0.0315388 0.0406668 2022-03-08T23:03:46.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 214 16 84106 84090 24.5558 18.875 0.0237835 0.0407121 2022-03-08T23:03:47.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 215 16 84353 84337 24.5134 15.4375 0.0755152 0.0407854 2022-03-08T23:03:48.307 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 216 16 84616 84600 24.476 16.4375 0.0407973 0.0408356 2022-03-08T23:03:49.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 217 16 84817 84801 24.4211 12.5625 0.17371 0.0409357 2022-03-08T23:03:50.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 218 16 85115 85099 24.3945 18.625 0.0835405 0.04098 2022-03-08T23:03:51.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 219 16 85416 85400 24.369 18.8125 0.0326302 0.0410259 2022-03-08T23:03:52.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:03:52.308629+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0410927 2022-03-08T23:03:52.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:03:52.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 220 16 85667 85651 24.3296 15.6875 0.111087 0.0410927 2022-03-08T23:03:53.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 221 16 85834 85818 24.2667 10.4375 0.0325892 0.0411269 2022-03-08T23:03:54.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 222 16 86130 86114 24.2407 18.5 0.0455193 0.0412415 2022-03-08T23:03:55.177 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-08T23:03:55.178 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:03:55.178 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:03:55.179 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-08T23:03:55.179 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:03:55.179 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-08T23:03:55.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 223 16 86398 86382 24.2071 16.75 0.0172849 0.0412951 2022-03-08T23:03:55.323 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:03:56.308 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 224 16 86655 86639 24.1708 16.0625 0.107245 0.0413569 2022-03-08T23:03:56.446 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=4 2022-03-08T23:03:56.446 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-08T23:03:57.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 225 16 86878 86862 24.1253 13.9375 0.0719877 0.0414367 2022-03-08T23:03:58.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 226 16 87136 87120 24.0899 16.125 0.0375504 0.0415014 2022-03-08T23:03:59.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 227 16 87369 87353 24.0479 14.5625 0.0475918 0.0415713 2022-03-08T23:04:00.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 228 16 87631 87615 24.0142 16.375 0.0599509 0.0416297 2022-03-08T23:04:01.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 229 16 87842 87826 23.9669 13.1875 0.0796663 0.0417137 2022-03-08T23:04:02.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 230 16 88140 88124 23.9437 18.625 0.0669602 0.0417537 2022-03-08T23:04:03.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 231 16 88442 88426 23.9218 18.875 0.0639865 0.0417917 2022-03-08T23:04:04.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 232 16 88540 88524 23.845 6.125 0.0472424 0.0419255 2022-03-08T23:04:05.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 233 16 88843 88827 23.824 18.9375 0.0166788 0.0419647 2022-03-08T23:04:06.309 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 234 16 89144 89128 23.8026 18.8125 0.053569 0.0420027 2022-03-08T23:04:07.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 235 16 89419 89403 23.7744 17.1875 0.0554615 0.0420514 2022-03-08T23:04:08.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 236 16 89692 89676 23.7459 17.0625 0.0793947 0.0421017 2022-03-08T23:04:09.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 237 16 89991 89975 23.7246 18.6875 0.0364065 0.042138 2022-03-08T23:04:10.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 238 16 90278 90262 23.7003 17.9375 0.0959692 0.0421831 2022-03-08T23:04:11.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 239 16 90572 90556 23.678 18.375 0.0347012 0.0422221 2022-03-08T23:04:12.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:04:12.310714+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0422928 2022-03-08T23:04:12.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:04:12.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 240 16 90796 90780 23.6377 14 0.00914351 0.0422928 2022-03-08T23:04:13.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 241 16 91094 91078 23.6168 18.625 0.0288214 0.0423275 2022-03-08T23:04:14.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 242 16 91378 91362 23.5926 17.75 0.0876513 0.0423707 2022-03-08T23:04:15.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 243 16 91573 91557 23.5457 12.1875 0.0374305 0.0424428 2022-03-08T23:04:16.310 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 244 16 91852 91836 23.5206 17.4375 0.023574 0.0425032 2022-03-08T23:04:17.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 245 16 92122 92106 23.4935 16.875 0.0964149 0.0425514 2022-03-08T23:04:18.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 246 16 92391 92375 23.4663 16.8125 0.0620307 0.0426036 2022-03-08T23:04:19.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 247 16 92682 92666 23.4449 18.1875 0.0323177 0.0426423 2022-03-08T23:04:20.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 248 16 92973 92957 23.4237 18.1875 0.0780163 0.0426815 2022-03-08T23:04:21.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 249 16 93260 93244 23.4017 17.9375 0.0360956 0.0427211 2022-03-08T23:04:22.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 250 16 93547 93531 23.3798 17.9375 0.0380641 0.0427616 2022-03-08T23:04:23.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 251 16 93822 93806 23.3552 17.1875 0.0888185 0.0428071 2022-03-08T23:04:24.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 252 16 94090 94074 23.3289 16.75 0.136392 0.0428499 2022-03-08T23:04:25.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 253 16 94308 94292 23.2906 13.625 0.0108695 0.0429011 2022-03-08T23:04:26.311 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 254 16 94533 94517 23.2542 14.0625 0.079073 0.0429929 2022-03-08T23:04:26.447 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 4 '!=' 4 ']' 2022-03-08T23:04:26.448 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:04:26.461 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch status 2022-03-08T23:04:27.312 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 255 16 94782 94766 23.2241 15.5625 0.0706296 0.0430414 2022-03-08T23:04:28.312 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 256 16 95051 95035 23.199 16.8125 0.106827 0.0430955 2022-03-08T23:04:29.315 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 257 16 95332 95316 23.1768 17.5625 0.0355553 0.0431336 2022-03-08T23:04:30.315 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 258 16 95602 95586 23.1523 16.875 0.0260713 0.0431778 2022-03-08T23:04:31.248 INFO:teuthology.orchestra.run.smithi186.stdout:Backend: rook 2022-03-08T23:04:31.296 INFO:teuthology.orchestra.run.smithi186.stdout:Available: Yes 2022-03-08T23:04:31.297 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ps 2022-03-08T23:04:31.315 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 259 16 95792 95776 23.1088 11.875 0.0253405 0.043215 2022-03-08T23:04:32.315 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:04:32.316301+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0433192 2022-03-08T23:04:32.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:04:32.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 260 16 96040 96024 23.0795 15.5 0.0763268 0.0433192 2022-03-08T23:04:32.479 INFO:teuthology.orchestra.run.smithi186.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:04:32.479 INFO:teuthology.orchestra.run.smithi186.stdout:crashcollector.crash smithi186 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-08T23:04:32.479 INFO:teuthology.orchestra.run.smithi186.stdout:mgr.a smithi186 running (9m) 0s ago 9m - - 1c5288aef625 2022-03-08T23:04:32.480 INFO:teuthology.orchestra.run.smithi186.stdout:mon.a smithi186 running (9m) 0s ago 9m - - 1c5288aef625 2022-03-08T23:04:32.480 INFO:teuthology.orchestra.run.smithi186.stdout:osd.0 smithi186 running (109s) 0s ago 109s - - 1c5288aef625 2022-03-08T23:04:32.480 INFO:teuthology.orchestra.run.smithi186.stdout:osd.1 smithi186 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-08T23:04:32.480 INFO:teuthology.orchestra.run.smithi186.stdout:osd.2 smithi186 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-08T23:04:32.481 INFO:teuthology.orchestra.run.smithi186.stdout:osd.3 smithi186 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-08T23:04:32.530 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls 2022-03-08T23:04:33.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 261 16 96258 96242 23.0433 13.625 0.138504 0.04337 2022-03-08T23:04:34.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 262 16 96499 96483 23.0128 15.0625 0.090836 0.0434404 2022-03-08T23:04:35.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 263 16 96710 96694 22.9755 13.1875 0.0912673 0.0435116 2022-03-08T23:04:35.667 INFO:teuthology.orchestra.run.smithi186.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:04:35.667 INFO:teuthology.orchestra.run.smithi186.stdout:crash 1/1 2s ago 7m * 2022-03-08T23:04:35.668 INFO:teuthology.orchestra.run.smithi186.stdout:mgr 1/1 2s ago 9m count:1 2022-03-08T23:04:35.668 INFO:teuthology.orchestra.run.smithi186.stdout:mon 1/1 2s ago 9m count:1 2022-03-08T23:04:35.668 INFO:teuthology.orchestra.run.smithi186.stdout:osd 8 2s ago 7m 2022-03-08T23:04:35.668 INFO:teuthology.orchestra.run.smithi186.stdout:osd.all-available-devices 0 2s ago - * 2022-03-08T23:04:35.724 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch host ls 2022-03-08T23:04:36.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 264 16 96959 96943 22.9474 15.5625 0.0323551 0.0435681 2022-03-08T23:04:37.290 INFO:teuthology.orchestra.run.smithi186.stdout:HOST ADDR LABELS STATUS 2022-03-08T23:04:37.291 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 172.21.15.186/smithi186 2022-03-08T23:04:37.291 INFO:teuthology.orchestra.run.smithi186.stdout:1 hosts in cluster 2022-03-08T23:04:37.306 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch device ls 2022-03-08T23:04:37.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 265 16 97047 97031 22.8815 5.5 0.00855665 0.0436079 2022-03-08T23:04:38.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 266 16 97056 97040 22.7976 0.5625 1.55114 0.0437475 2022-03-08T23:04:38.354 INFO:teuthology.orchestra.run.smithi186.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:04:38.355 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:04:38.355 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:04:38.355 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:04:38.355 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:04:38.394 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch apply rgw foo 2022-03-08T23:04:39.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 267 16 97281 97265 22.7649 14.0625 0.155901 0.043916 2022-03-08T23:04:39.883 INFO:teuthology.orchestra.run.smithi186.stdout:Created 2022-03-08T23:04:39.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:39.887174 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T23:04:39.941 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch apply mds foo 2022-03-08T23:04:39.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:39.940371 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:04:40.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 268 16 97504 97488 22.732 13.9375 0.0608149 0.0439784 2022-03-08T23:04:41.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:41.015258 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:04:41.019 INFO:teuthology.orchestra.run.smithi186.stdout:Created 2022-03-08T23:04:41.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:41.066028 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:04:41.067 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch apply rbd-mirror 2022-03-08T23:04:41.086 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:41.082104 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:04:41.316 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 269 16 97736 97720 22.7014 14.5 0.0583097 0.0440368 2022-03-08T23:04:42.199 INFO:teuthology.orchestra.run.smithi186.stdout:Success 2022-03-08T23:04:42.257 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch apply nfs foo --port 12777 2022-03-08T23:04:42.294 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:42.290504 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:04:42.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 270 16 97926 97910 22.6613 11.875 0.131393 0.0441134 2022-03-08T23:04:42.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:42.643865 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:42.742 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:42.742730 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:04:42.748 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:42.743148 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:43.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 271 16 98085 98069 22.6143 9.9375 0.160829 0.0442055 2022-03-08T23:04:44.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:44.072837 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:44.122 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:44.121412 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T23:04:44.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:44.121440 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T23:04:44.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:44.121452 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T23:04:44.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 272 16 98276 98260 22.5751 11.9375 0.0972711 0.0442798 2022-03-08T23:04:45.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 273 16 98394 98378 22.5194 7.375 0.294878 0.0443772 2022-03-08T23:04:45.830 INFO:teuthology.orchestra.run.smithi186.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T23:04:45.830 INFO:teuthology.orchestra.run.smithi186.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T23:04:45.872 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-08T23:04:45.883 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-08T23:04:45.883 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-08T23:04:45.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:45.950054 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T23:04:46.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 274 16 98544 98528 22.4714 9.375 0.0558387 0.0444456 2022-03-08T23:04:46.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:46.883827 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:04:47.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 275 16 98690 98674 22.4229 9.125 0.123709 0.0445743 2022-03-08T23:04:48.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 276 16 98871 98855 22.3826 11.3125 0.0289432 0.0446571 2022-03-08T23:04:48.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:04:48.368035 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T23:04:49.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 277 16 99092 99076 22.3517 13.8125 0.0221127 0.0447273 2022-03-08T23:04:50.317 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 278 16 99276 99260 22.3126 11.5 0.0643837 0.044785 2022-03-08T23:04:51.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 279 16 99475 99459 22.2772 12.4375 0.0223802 0.0448746 2022-03-08T23:04:52.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:04:52.318369+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0449334 2022-03-08T23:04:52.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:04:52.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 280 16 99708 99692 22.2497 14.5625 0.061123 0.0449334 2022-03-08T23:04:53.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 281 16 99930 99914 22.2199 13.875 0.0906616 0.0449928 2022-03-08T23:04:54.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 282 16 100143 100127 22.1883 13.3125 0.106672 0.0450532 2022-03-08T23:04:55.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 283 16 100359 100343 22.1576 13.5 0.108229 0.0451163 2022-03-08T23:04:56.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 284 16 100535 100519 22.1183 11 0.0885944 0.045197 2022-03-08T23:04:57.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 285 16 100718 100702 22.0808 11.4375 0.0351553 0.0452774 2022-03-08T23:04:58.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 286 16 100972 100956 22.0591 15.875 0.0921652 0.0453209 2022-03-08T23:04:59.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 287 16 101125 101109 22.0156 9.5625 0.0836928 0.0454114 2022-03-08T23:05:00.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:00.255881 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-08T23:05:00.298 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:00.262844 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-08T23:05:00.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:00.278572 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-08T23:05:00.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:00.287218 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-08T23:05:00.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:00.287464 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-08T23:05:00.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:00.302796 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-08T23:05:00.318 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 288 16 101380 101364 21.9944 15.9375 0.155514 0.0454476 2022-03-08T23:05:01.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 289 16 101533 101517 21.9514 9.5625 0.104645 0.045536 2022-03-08T23:05:02.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 290 16 101682 101666 21.9078 9.3125 0.0783768 0.0456297 2022-03-08T23:05:03.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 291 16 101939 101923 21.8878 16.0625 0.0382353 0.0456763 2022-03-08T23:05:04.246 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:04.245985 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-08T23:05:04.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 292 16 102176 102160 21.8635 14.8125 0.114933 0.045711 2022-03-08T23:05:05.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 293 16 102350 102334 21.826 10.875 0.0566045 0.0458049 2022-03-08T23:05:06.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 294 16 102610 102594 21.807 16.25 0.0360112 0.0458456 2022-03-08T23:05:06.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:06.886910 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T23:05:07.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 295 16 102854 102838 21.7848 15.25 0.0371549 0.0458913 2022-03-08T23:05:08.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.049573 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-08T23:05:08.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.087687 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-08T23:05:08.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.094783 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-08T23:05:08.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.123080 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-08T23:05:08.196 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.193749 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:08.196 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.193774 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:08.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 296 16 103084 103068 21.7598 14.375 0.0973405 0.0459447 2022-03-08T23:05:08.468 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:08.468387 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:05:09.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 297 16 103289 103273 21.7296 12.8125 0.053621 0.046009 2022-03-08T23:05:10.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 298 16 103500 103484 21.701 13.1875 0.0384452 0.0460612 2022-03-08T23:05:11.319 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 299 16 103749 103733 21.6804 15.5625 0.0553605 0.0461128 2022-03-08T23:05:12.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:05:12.320407+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.046158 2022-03-08T23:05:12.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:05:12.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 300 16 103999 103983 21.6602 15.625 0.0518516 0.046158 2022-03-08T23:05:12.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:12.489824 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:12.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:12.807065 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-08T23:05:12.820 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:12.819075 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:05:12.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:12.819099 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:13.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 301 16 104213 104197 21.6327 13.375 0.122009 0.0462149 2022-03-08T23:05:13.624 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:13.622718 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:14.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 302 16 104403 104387 21.6004 11.875 0.068513 0.0462829 2022-03-08T23:05:14.551 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:14.550869 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-08T23:05:14.551 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:14.550907 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T23:05:15.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 303 16 104565 104549 21.5625 10.125 0.0436198 0.046353 2022-03-08T23:05:16.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 304 16 104789 104773 21.5377 14 0.0323365 0.046421 2022-03-08T23:05:17.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:17.213818 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T23:05:17.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 305 16 104998 104982 21.5099 13.0625 0.178405 0.0464805 2022-03-08T23:05:18.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:17.929690 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T23:05:18.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 306 16 105165 105149 21.4737 10.4375 0.0918937 0.0465561 2022-03-08T23:05:18.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:18.537139 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:18.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:18.562954 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:05:19.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 307 16 105421 105405 21.4558 16 0.0695514 0.0465935 2022-03-08T23:05:19.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:19.371538 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T23:05:20.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:20.284774 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T23:05:20.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 308 16 105630 105614 21.4286 13.0625 0.0261566 0.046655 2022-03-08T23:05:21.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:21.035109 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:21.196 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:21.196081 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:05:21.196 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:21.196107 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:21.320 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 309 16 105873 105857 21.4084 15.1875 0.172983 0.0466979 2022-03-08T23:05:21.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:21.402453 I | op-mds: setting mds config flags 2022-03-08T23:05:21.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:21.402489 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:05:21.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:21.893550 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:22.321 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 310 16 106081 106065 21.3812 13 0.0830612 0.0467564 2022-03-08T23:05:22.417 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:22.415670 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:05:22.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:22.425927 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T23:05:23.321 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 311 16 106226 106210 21.3416 9.0625 0.0553765 0.0468388 2022-03-08T23:05:24.321 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 312 16 106478 106462 21.3237 15.75 0.0353939 0.0468831 2022-03-08T23:05:24.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:24.668040 I | op-mds: setting mds config flags 2022-03-08T23:05:24.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:24.668080 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:05:25.321 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 313 16 106691 106675 21.2981 13.3125 0.0277994 0.0469393 2022-03-08T23:05:26.151 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:26.151412 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:26.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:26.316263 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:05:26.321 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 314 16 106912 106896 21.2742 13.8125 0.0336661 0.0469907 2022-03-08T23:05:26.341 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:26.339248 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:05:27.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 315 16 107137 107121 21.2513 14.0625 0.0936009 0.0470446 2022-03-08T23:05:28.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 316 16 107297 107281 21.2157 10 0.045383 0.0471216 2022-03-08T23:05:29.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 317 16 107520 107504 21.1928 13.9375 0.0267496 0.0471719 2022-03-08T23:05:30.084 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:30.083102 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:30.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 318 16 107755 107739 21.1723 14.6875 0.0304129 0.0472212 2022-03-08T23:05:31.249 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:31.248798 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:05:31.250 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:31.248819 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:31.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 319 16 107982 107966 21.1504 14.1875 0.0653536 0.0472698 2022-03-08T23:05:31.923 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:31.922904 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:32.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:05:32.323001+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0473414 2022-03-08T23:05:32.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:05:32.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 320 16 108159 108143 21.1189 11.0625 0.0421741 0.0473414 2022-03-08T23:05:33.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 321 16 108362 108346 21.0926 12.6875 0.0331989 0.0473973 2022-03-08T23:05:34.322 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 322 16 108571 108555 21.0677 13.0625 0.0224507 0.0474549 2022-03-08T23:05:35.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 323 16 108795 108779 21.0458 14 0.0478671 0.0474936 2022-03-08T23:05:36.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 324 16 109048 109032 21.0296 15.8125 0.0799527 0.0475408 2022-03-08T23:05:36.392 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:36.392511 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:36.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:36.501841 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:05:37.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 325 16 109275 109259 21.0086 14.1875 0.0182233 0.0475872 2022-03-08T23:05:38.323 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 326 16 109500 109484 20.9873 14.0625 0.0439268 0.0476326 2022-03-08T23:05:39.024 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:39.024237 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:39.326 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 327 16 109729 109713 20.9666 14.3125 0.0607045 0.0476832 2022-03-08T23:05:40.326 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 328 16 109948 109932 20.9444 13.6875 0.115968 0.047733 2022-03-08T23:05:40.782 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:40.779292 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:05:40.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:40.779443 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:41.326 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 329 16 110182 110166 20.9252 14.625 0.0637213 0.0477761 2022-03-08T23:05:41.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:41.491292 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:42.326 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 330 16 110321 110305 20.8882 8.6875 0.153435 0.0478575 2022-03-08T23:05:43.326 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 331 16 110525 110509 20.8636 12.75 0.172526 0.0479132 2022-03-08T23:05:44.326 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 332 16 110755 110739 20.844 14.375 0.20035 0.04796 2022-03-08T23:05:45.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 333 16 110988 110972 20.8251 14.5625 0.112033 0.0480045 2022-03-08T23:05:46.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:46.000350 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:46.262 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:46.262115 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:05:46.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 334 16 111220 111204 20.8062 14.5 0.103392 0.0480515 2022-03-08T23:05:47.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 335 16 111434 111418 20.784 13.375 0.0262986 0.0481038 2022-03-08T23:05:48.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 336 16 111669 111653 20.7659 14.6875 0.0822663 0.0481451 2022-03-08T23:05:48.598 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:48.595183 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:48.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:48.709214 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:05:48.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:48.709248 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:49.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 337 16 111923 111907 20.7514 15.875 0.0643505 0.0481804 2022-03-08T23:05:49.626 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:49.622669 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:50.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 338 16 112098 112082 20.7223 10.9375 0.150453 0.0482397 2022-03-08T23:05:51.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 339 16 112318 112302 20.7017 13.75 0.0930509 0.0482949 2022-03-08T23:05:52.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-08T23:05:52.328122+0000 min lat: 0.00169504 max lat: 11.1655 avg lat: 0.0483418 2022-03-08T23:05:52.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:05:52.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 340 16 112537 112521 20.6811 13.6875 0.110319 0.0483418 2022-03-08T23:05:53.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:53.051254 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:53.279 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:53.277985 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:05:53.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 341 16 112760 112744 20.6613 13.9375 0.051525 0.0483892 2022-03-08T23:05:54.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 342 16 113010 112994 20.6466 15.625 0.0544885 0.048424 2022-03-08T23:05:55.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 343 16 113235 113219 20.6274 14.0625 0.0382284 0.0484662 2022-03-08T23:05:55.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:55.733131 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:55.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:55.822688 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:05:55.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:55.822842 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:56.327 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 344 16 113464 113448 20.609 14.3125 0.0706939 0.0485119 2022-03-08T23:05:56.542 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:05:56.538523 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:57.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 345 16 113631 113615 20.5796 10.4375 0.116456 0.04858 2022-03-08T23:05:58.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 346 16 113817 113801 20.5537 11.625 0.160228 0.0486427 2022-03-08T23:05:59.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 347 16 114062 114046 20.5386 15.3125 0.0372482 0.0486782 2022-03-08T23:06:00.194 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:00.194102 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:00.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 348 16 114298 114282 20.5219 14.75 0.134201 0.0487161 2022-03-08T23:06:00.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:00.521118 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:06:01.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 349 16 114552 114536 20.5086 15.875 0.0900805 0.0487487 2022-03-08T23:06:02.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 350 16 114806 114790 20.4954 15.875 0.0601733 0.0487789 2022-03-08T23:06:03.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:03.246619 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:03.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 351 16 114892 114876 20.4523 5.375 0.0193983 0.0488618 2022-03-08T23:06:03.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:03.516881 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:06:03.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:03.516906 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:04.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:04.142775 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:06:04.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 352 16 115115 115099 20.4338 13.9375 0.0816494 0.0489238 2022-03-08T23:06:05.328 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 353 16 115249 115233 20.3996 8.375 0.0101216 0.0489895 2022-03-08T23:06:06.329 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 354 16 115342 115326 20.3584 5.8125 0.0297886 0.049109 2022-03-08T23:06:07.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:07.144834 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:07.329 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 355 16 115574 115558 20.3419 14.5 0.147337 0.0491493 2022-03-08T23:06:07.793 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:07.790882 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:06:08.329 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 356 16 115873 115857 20.3373 18.6875 0.178107 0.0491533 2022-03-08T23:06:09.329 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 357 16 116118 116102 20.3232 15.3125 0.0708689 0.049191 2022-03-08T23:06:09.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:09.844494 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:09.932 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:09.932195 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:06:09.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:09.932220 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:10.329 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 358 16 116384 116368 20.3128 16.625 0.0401873 0.0492202 2022-03-08T23:06:10.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:10.546235 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:06:11.329 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 359 16 116641 116625 20.301 16.0625 0.122593 0.0492452 2022-03-08T23:06:12.330 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Total time run: 360.034 2022-03-08T23:06:12.330 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Total writes made: 116924 2022-03-08T23:06:12.330 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Write size: 65536 2022-03-08T23:06:12.330 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Object size: 65536 2022-03-08T23:06:12.331 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Bandwidth (MB/sec): 20.2974 2022-03-08T23:06:12.331 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Stddev Bandwidth: 14.7006 2022-03-08T23:06:12.331 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Max bandwidth (MB/sec): 95.375 2022-03-08T23:06:12.331 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Min bandwidth (MB/sec): 0 2022-03-08T23:06:12.332 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Average IOPS: 324 2022-03-08T23:06:12.332 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Stddev IOPS: 235.22 2022-03-08T23:06:12.332 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Max IOPS: 1526 2022-03-08T23:06:12.332 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Min IOPS: 0 2022-03-08T23:06:12.333 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Average Latency(s): 0.0492595 2022-03-08T23:06:12.333 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Stddev Latency(s): 0.245313 2022-03-08T23:06:12.333 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Max latency(s): 11.1655 2022-03-08T23:06:12.333 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Min latency(s): 0.00169504 2022-03-08T23:06:12.334 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Cleaning up (deleting benchmark objects) 2022-03-08T23:06:13.442 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:13.442264 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:15.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:15.054739 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:06:17.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:17.371076 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:18.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:18.758403 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:06:18.990 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:18.758430 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:19.334 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:19.322606 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:06:23.308 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:23.307159 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:25.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:25.881258 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:06:28.249 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:28.249332 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:29.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:29.237976 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:06:29.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:29.237998 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:29.791 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:29.789239 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:06:32.658 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:32.658639 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:37.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:37.791375 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:06:39.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:39.981664 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:40.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:40.142198 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:06:40.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:40.142224 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:40.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:40.721619 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:06:43.598 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:43.598139 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:53.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:53.845115 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:06:56.265 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:56.263191 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:58.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:58.126771 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:06:58.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:58.126926 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:59.339 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:06:59.279991 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:07:03.366 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:03.365875 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:07:23.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:23.857188 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:07:26.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:26.093928 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:07:26.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:26.575540 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:07:26.579 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:26.575566 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:07:27.289 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:27.287634 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:07:30.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:07:30.874601 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:08:11.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:08:11.844707 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:08:13.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:08:13.953861 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:08:14.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:08:14.618380 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:08:14.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:08:14.618401 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:08:15.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:08:15.277826 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:08:19.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:08:19.081371 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:09:41.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:09:41.015526 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:09:43.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:09:43.193573 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:09:43.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:09:43.836016 I | ceph-object-controller: ceph object store gateway service running at 10.111.103.192 2022-03-08T23:09:43.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:09:43.836043 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:09:44.417 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:09:44.417464 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:09:48.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:09:48.380998 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:13.055 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Removed 116924 objects 2022-03-08T23:12:13.056 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Clean up completed and total clean up time :360.721 2022-03-08T23:12:17.376 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-08T23:12:17.377 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-08T23:12:18.244 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'unique_pool_0' removed 2022-03-08T23:12:18.260 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-08T23:12:18.275 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-08T23:12:18.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-08T23:12:18.290 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T23:12:18.313 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:12:19.052 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:12:19.052 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:04:46.000000Z", "last_refresh": "2022-03-08T23:12:18.967926Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:18.967926Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:12:18.967926Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:12:18.967926Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:12:18.967926Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:18.967926Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T23:12:18.967926Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:18.967926Z", "running": 0, "size": 1}}] 2022-03-08T23:12:19.073 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch rm mds.foo 2022-03-08T23:12:19.678 INFO:teuthology.orchestra.run.smithi186.stdout:Removed cephfilesystems/foo 2022-03-08T23:12:19.697 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch rm rbd-mirror 2022-03-08T23:12:19.713 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:19.713244 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:12:20.220 INFO:teuthology.orchestra.run.smithi186.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T23:12:20.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:20.236736 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T23:12:20.246 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch rm rgw.foo 2022-03-08T23:12:21.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:21.095515 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:Removed cephobjectstores/foo 2022-03-08T23:12:21.129 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:12:21.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:21.450420 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:12:22.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:22.759095 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:22.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:22.759116 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T23:12:22.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:22.759122 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T23:12:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:12:23.995 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:04:46.000000Z", "last_refresh": "2022-03-08T23:12:21.933558Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:21.933558Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:12:21.933558Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:12:21.933558Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:12:21.933558Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:21.933558Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:21.933558Z", "running": 0, "size": 1}}] 2022-03-08T23:12:25.452 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:25.451455 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T23:12:26.106 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:26.106172 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:27.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:27.275442 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:27.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:27.275473 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T23:12:27.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:27.275480 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T23:12:27.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:27.963631 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:27.945+0000 7f8e5f47c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:27.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:27.945+0000 7f8e5f47c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:27.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:27.945+0000 7f8e5f47c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:27.966 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:27.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:27.963659 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:27.945+0000 7f8e5f47c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:27.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:27.945+0000 7f8e5f47c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:27.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:27.945+0000 7f8e5f47c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:27.967 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:28.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:28.439021 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T23:12:28.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:28.439053 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T23:12:31.212 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:31.212799 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T23:12:32.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:32.861930 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:12:33.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:33.485235 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:33.477+0000 7f58318e8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:33.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:33.477+0000 7f58318e8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:33.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:33.477+0000 7f58318e8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:33.486 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:33.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:33.485256 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:33.477+0000 7f58318e8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:33.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:33.477+0000 7f58318e8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:33.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:12:33.477+0000 7f58318e8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:33.487 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:33.595 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:33.596123 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T23:12:33.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:33.596143 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T23:12:34.026 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:12:35.749 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:12:35.749 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:04:46.000000Z", "last_refresh": "2022-03-08T23:12:35.375993Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:35.375993Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:12:35.375993Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:12:35.375993Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:12:35.375993Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:35.375993Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:35.375993Z", "running": 0, "size": 1}}] 2022-03-08T23:12:37.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:37.566237 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T23:12:37.993 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:37.993677 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T23:12:37.993 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:37.993694 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T23:12:40.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:40.446279 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T23:12:40.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:40.446322 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:12:40.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:40.463521 E | ceph-file-controller: failed to reconcile failed to remove finalizer: failed to remove finalizer "cephfilesystem.ceph.rook.io" on "foo": Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:40.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:40.473071 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:12:41.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:41.281344 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:12:41.338 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:41.338191 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T23:12:41.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:41.344567 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T23:12:41.351 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:41.346999 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T23:12:41.351 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:41.347340 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T23:12:45.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:45.244250 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:45.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:12:45.693606 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:45.772 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:12:46.775 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:12:46.776 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:04:46.000000Z", "last_refresh": "2022-03-08T23:12:46.280862Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:12:46.280862Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:12:46.280862Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:12:46.280862Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:46.280862Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:46.280862Z", "running": 0, "size": 1}}] 2022-03-08T23:12:56.828 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:12:58.528 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:12:58.529 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:04:46.000000Z", "last_refresh": "2022-03-08T23:12:57.463007Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:12:57.463007Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:12:57.463007Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:12:57.463007Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:57.463007Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:57.463007Z", "running": 0, "size": 1}}] 2022-03-08T23:13:08.555 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:13:09.151 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:13:09.151 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:13:09.053104Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:13:09.053104Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:13:09.053104Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:13:09.053104Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:09.053104Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:09.053104Z", "running": 0, "size": 1}}] 2022-03-08T23:13:19.171 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:13:19.766 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:13:19.766 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:13:19.667405Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:13:19.667405Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:13:19.667405Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:13:19.667405Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:19.667405Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:19.667405Z", "running": 0, "size": 1}}] 2022-03-08T23:13:29.792 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:13:31.270 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:13:31.271 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:13:30.530474Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:13:30.530474Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:13:30.530474Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:13:30.530474Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:30.530474Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:30.530474Z", "running": 0, "size": 1}}] 2022-03-08T23:13:41.295 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:13:41.897 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:13:41.898 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:13:41.803762Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:13:41.803762Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:13:41.803762Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:13:41.803762Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:41.803762Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:41.803762Z", "running": 0, "size": 1}}] 2022-03-08T23:13:51.920 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:13:52.569 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:13:52.570 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:13:52.466571Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:13:52.466571Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:13:52.466571Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:13:52.466571Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:52.466571Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:52.466571Z", "running": 0, "size": 1}}] 2022-03-08T23:14:02.592 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:14:03.414 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:14:03.415 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:14:03.289457Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:14:03.289457Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:14:03.289457Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:14:03.289457Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:03.289457Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:03.289457Z", "running": 0, "size": 1}}] 2022-03-08T23:14:13.438 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:14:14.085 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:14:14.086 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:14:13.984866Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:14:13.984866Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:14:13.984866Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:14:13.984866Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:13.984866Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:13.984866Z", "running": 0, "size": 1}}] 2022-03-08T23:14:24.108 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:14:24.751 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:14:24.751 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:14:24.604763Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:14:24.604763Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:14:24.604763Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:14:24.604763Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:24.604763Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:24.604763Z", "running": 0, "size": 1}}] 2022-03-08T23:14:34.775 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:14:38.680 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:14:38.681 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:14:38.544805Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:14:38.544805Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:14:38.544805Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:14:38.544805Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:38.544805Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:38.544805Z", "running": 0, "size": 1}}] 2022-03-08T23:14:48.703 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:14:49.361 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:14:49.362 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:14:49.251456Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:14:49.251456Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:14:49.251456Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:14:49.251456Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:49.251456Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:49.251456Z", "running": 0, "size": 1}}] 2022-03-08T23:14:59.381 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:15:00.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:15:00.075 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:14:59.915970Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:14:59.915970Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:14:59.915970Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:14:59.915970Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:59.915970Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:59.915970Z", "running": 0, "size": 1}}] 2022-03-08T23:15:10.096 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:15:10.702 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:15:10.703 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:15:10.609386Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:15:10.609386Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:15:10.609386Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:15:10.609386Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:10.609386Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:10.609386Z", "running": 0, "size": 1}}] 2022-03-08T23:15:20.727 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:15:21.340 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:15:21.340 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:15:21.246774Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:15:21.246774Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:15:21.246774Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:15:21.246774Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:21.246774Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:21.246774Z", "running": 0, "size": 1}}] 2022-03-08T23:15:31.365 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:15:32.348 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:15:32.348 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:15:31.972226Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:15:31.972226Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:15:31.972226Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:15:31.972226Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:31.972226Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:31.972226Z", "running": 0, "size": 1}}] 2022-03-08T23:15:42.372 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:15:43.020 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:15:43.020 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:15:42.923646Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:15:42.923646Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:15:42.923646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:15:42.923646Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:42.923646Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:42.923646Z", "running": 0, "size": 1}}] 2022-03-08T23:15:53.044 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:15:53.669 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:15:53.669 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:15:53.571379Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:15:53.571379Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:15:53.571379Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:15:53.571379Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:53.571379Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:53.571379Z", "running": 0, "size": 1}}] 2022-03-08T23:16:03.695 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:16:04.380 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:16:04.381 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:16:04.254780Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:16:04.254780Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:16:04.254780Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:16:04.254780Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:04.254780Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:04.254780Z", "running": 0, "size": 1}}] 2022-03-08T23:16:14.408 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:16:15.031 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:16:15.031 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:16:14.920834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:16:14.920834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:16:14.920834Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:16:14.920834Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:14.920834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:14.920834Z", "running": 0, "size": 1}}] 2022-03-08T23:16:25.052 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:16:25.755 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:16:25.755 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:16:25.659763Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:16:25.659763Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:16:25.659763Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:16:25.659763Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:25.659763Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:25.659763Z", "running": 0, "size": 1}}] 2022-03-08T23:16:35.780 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:16:36.425 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:16:36.426 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:16:36.312700Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:16:36.312700Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:16:36.312700Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:16:36.312700Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:36.312700Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:36.312700Z", "running": 0, "size": 1}}] 2022-03-08T23:16:46.468 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:16:47.173 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:16:47.173 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:16:47.055195Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:16:47.055195Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:16:47.055195Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:16:47.055195Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:47.055195Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:47.055195Z", "running": 0, "size": 1}}] 2022-03-08T23:16:57.196 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:16:57.928 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:16:57.928 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:16:57.720103Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:16:57.720103Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:16:57.720103Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:16:57.720103Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:57.720103Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:57.720103Z", "running": 0, "size": 1}}] 2022-03-08T23:17:07.951 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:17:08.706 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:17:08.706 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:17:08.588299Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:17:08.588299Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:17:08.588299Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:17:08.588299Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:08.588299Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:08.588299Z", "running": 0, "size": 1}}] 2022-03-08T23:17:18.726 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:17:19.370 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:17:19.370 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:17:19.288966Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:17:19.288966Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:17:19.288966Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:17:19.288966Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:19.288966Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:19.288966Z", "running": 0, "size": 1}}] 2022-03-08T23:17:29.389 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:17:30.001 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:17:30.001 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:17:29.901595Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:17:29.901595Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:17:29.901595Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:17:29.901595Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:29.901595Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:29.901595Z", "running": 0, "size": 1}}] 2022-03-08T23:17:40.025 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:17:40.659 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:17:40.659 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:17:40.567257Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:17:40.567257Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:17:40.567257Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:17:40.567257Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:40.567257Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:40.567257Z", "running": 0, "size": 1}}] 2022-03-08T23:17:50.681 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:17:51.375 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:17:51.375 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:17:51.207498Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:17:51.207498Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:17:51.207498Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:17:51.207498Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:51.207498Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:51.207498Z", "running": 0, "size": 1}}] 2022-03-08T23:18:01.401 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:18:02.127 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:18:02.128 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:18:01.991741Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:18:01.991741Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:18:01.991741Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:18:01.991741Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:01.991741Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:01.991741Z", "running": 0, "size": 1}}] 2022-03-08T23:18:12.149 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:18:12.676646Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:18:12.676646Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:18:12.676646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:18:12.676646Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:12.676646Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:12.676646Z", "running": 0, "size": 1}}] 2022-03-08T23:18:22.800 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:18:23.422 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:18:23.423 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:18:23.300560Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:18:23.300560Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:18:23.300560Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:18:23.300560Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:23.300560Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:23.300560Z", "running": 0, "size": 1}}] 2022-03-08T23:18:33.446 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:18:34.128 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:18:34.128 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:18:34.021298Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:18:34.021298Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:18:34.021298Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:18:34.021298Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:34.021298Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:34.021298Z", "running": 0, "size": 1}}] 2022-03-08T23:18:44.154 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:18:44.739 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:18:44.739 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:18:44.643565Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:18:44.643565Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:18:44.643565Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:18:44.643565Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:44.643565Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:44.643565Z", "running": 0, "size": 1}}] 2022-03-08T23:18:54.761 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:18:55.496 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:18:55.496 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:18:55.281026Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:18:55.281026Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:18:55.281026Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:18:55.281026Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:55.281026Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:55.281026Z", "running": 0, "size": 1}}] 2022-03-08T23:19:05.517 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:19:06.138 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:19:06.138 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:19:06.041486Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:19:06.041486Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:19:06.041486Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:19:06.041486Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:06.041486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:06.041486Z", "running": 0, "size": 1}}] 2022-03-08T23:19:16.161 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:19:16.812 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:19:16.812 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:19:16.688716Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:19:16.688716Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:19:16.688716Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:19:16.688716Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:16.688716Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:16.688716Z", "running": 0, "size": 1}}] 2022-03-08T23:19:26.837 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:19:27.452 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:19:27.452 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:19:27.360023Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:19:27.360023Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:19:27.360023Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:19:27.360023Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:27.360023Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:27.360023Z", "running": 0, "size": 1}}] 2022-03-08T23:19:37.482 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:19:38.117 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:19:38.117 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:19:38.035364Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:19:38.035364Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:19:38.035364Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:19:38.035364Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:38.035364Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:38.035364Z", "running": 0, "size": 1}}] 2022-03-08T23:19:48.139 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:19:48.876 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:19:48.876 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:19:48.763237Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:19:48.763237Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:19:48.763237Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:19:48.763237Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:48.763237Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:48.763237Z", "running": 0, "size": 1}}] 2022-03-08T23:19:58.899 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:19:59.477 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:19:59.478 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:19:59.390270Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:19:59.390270Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:19:59.390270Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:19:59.390270Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:59.390270Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:59.390270Z", "running": 0, "size": 1}}] 2022-03-08T23:20:09.497 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:20:10.275 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:20:10.276 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:20:10.157731Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:20:10.157731Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:20:10.157731Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:20:10.157731Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:10.157731Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:10.157731Z", "running": 0, "size": 1}}] 2022-03-08T23:20:20.305 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:20:20.961 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:20:20.962 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:20:20.856683Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:20:20.856683Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:20:20.856683Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:20:20.856683Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:20.856683Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:20.856683Z", "running": 0, "size": 1}}] 2022-03-08T23:20:30.984 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:20:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:20:31.593 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:20:31.501908Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:20:31.501908Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:20:31.501908Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:20:31.501908Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:31.501908Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:31.501908Z", "running": 0, "size": 1}}] 2022-03-08T23:20:41.615 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:20:42.248 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:20:42.248 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:20:42.152115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:20:42.152115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:20:42.152115Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:20:42.152115Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:42.152115Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:42.152115Z", "running": 0, "size": 1}}] 2022-03-08T23:20:52.272 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:20:52.926 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:20:52.926 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:20:52.806162Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:20:52.806162Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:20:52.806162Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:20:52.806162Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:52.806162Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:52.806162Z", "running": 0, "size": 1}}] 2022-03-08T23:21:02.951 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:21:03.571 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:21:03.572 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:21:03.482112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:21:03.482112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:21:03.482112Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:21:03.482112Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:03.482112Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:03.482112Z", "running": 0, "size": 1}}] 2022-03-08T23:21:13.595 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:21:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:21:14.355 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:21:14.255160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:21:14.255160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:21:14.255160Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:21:14.255160Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:14.255160Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:14.255160Z", "running": 0, "size": 1}}] 2022-03-08T23:21:24.378 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:21:25.018 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:21:25.018 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:21:24.920983Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:21:24.920983Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:21:24.920983Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:21:24.920983Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:24.920983Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:24.920983Z", "running": 0, "size": 1}}] 2022-03-08T23:21:35.042 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:21:35.692 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:21:35.692 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:21:35.597107Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:21:35.597107Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:21:35.597107Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:21:35.597107Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:35.597107Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:35.597107Z", "running": 0, "size": 1}}] 2022-03-08T23:21:45.718 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:21:46.323 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:21:46.323 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:21:46.228284Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:21:46.228284Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:21:46.228284Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:21:46.228284Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:46.228284Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:46.228284Z", "running": 0, "size": 1}}] 2022-03-08T23:21:56.347 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:21:56.923 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:21:56.923 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:21:56.835389Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:21:56.835389Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:21:56.835389Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:21:56.835389Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:56.835389Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:56.835389Z", "running": 0, "size": 1}}] 2022-03-08T23:22:06.944 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:22:07.572 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:22:07.573 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:22:07.458389Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:22:07.458389Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:22:07.458389Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:22:07.458389Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:07.458389Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:07.458389Z", "running": 0, "size": 1}}] 2022-03-08T23:22:17.595 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:22:18.175 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:22:18.175 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:22:18.073757Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:22:18.073757Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:22:18.073757Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:22:18.073757Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:18.073757Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:18.073757Z", "running": 0, "size": 1}}] 2022-03-08T23:22:28.198 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:22:28.811 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:22:28.812 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:22:28.717561Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:22:28.717561Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:22:28.717561Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:22:28.717561Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:28.717561Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:28.717561Z", "running": 0, "size": 1}}] 2022-03-08T23:22:38.837 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:22:39.495 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:22:39.496 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:22:39.406436Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:22:39.406436Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:22:39.406436Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:22:39.406436Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:39.406436Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:39.406436Z", "running": 0, "size": 1}}] 2022-03-08T23:22:49.519 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:22:50.164 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:22:50.165 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:22:50.033737Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:22:50.033737Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:22:50.033737Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:22:50.033737Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:50.033737Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:50.033737Z", "running": 0, "size": 1}}] 2022-03-08T23:23:00.188 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:23:00.846 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:23:00.847 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:23:00.748577Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:23:00.748577Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:23:00.748577Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:23:00.748577Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:00.748577Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:00.748577Z", "running": 0, "size": 1}}] 2022-03-08T23:23:10.873 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:23:11.527 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:23:11.528 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:23:11.435650Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:23:11.435650Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:23:11.435650Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:23:11.435650Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:11.435650Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:11.435650Z", "running": 0, "size": 1}}] 2022-03-08T23:23:21.554 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:23:22.178 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:23:22.178 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:23:22.080727Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:23:22.080727Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:23:22.080727Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:23:22.080727Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:22.080727Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:22.080727Z", "running": 0, "size": 1}}] 2022-03-08T23:23:28.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:23:28.853051 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:23:29.518 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:23:29.517837 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:23:29.508+0000 7f1a83360340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:23:29.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:23:29.508+0000 7f1a83360340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:23:29.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:23:29.508+0000 7f1a83360340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:23:29.520 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:23:29.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:23:29.517865 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:23:29.508+0000 7f1a83360340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:23:29.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:23:29.508+0000 7f1a83360340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:23:29.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:23:29.508+0000 7f1a83360340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:23:29.521 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:23:32.201 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:23:32.815 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:23:32.816 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:23:32.721775Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:23:32.721775Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:23:32.721775Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:23:32.721775Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:32.721775Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:32.721775Z", "running": 0, "size": 1}}] 2022-03-08T23:23:42.842 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:23:43.483 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:23:43.484 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:23:43.388721Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:23:43.388721Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:23:43.388721Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:23:43.388721Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:43.388721Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:43.388721Z", "running": 0, "size": 1}}] 2022-03-08T23:23:53.504 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:23:54.188 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:23:54.189 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:23:54.031194Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:23:54.031194Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:23:54.031194Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:23:54.031194Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:54.031194Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:54.031194Z", "running": 0, "size": 1}}] 2022-03-08T23:24:04.213 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:24:04.854 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:24:04.854 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:24:04.733267Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:24:04.733267Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:24:04.733267Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:24:04.733267Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:04.733267Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:04.733267Z", "running": 0, "size": 1}}] 2022-03-08T23:24:14.882 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:24:15.480 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:24:15.481 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:24:15.385442Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:24:15.385442Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:24:15.385442Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:24:15.385442Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:15.385442Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:15.385442Z", "running": 0, "size": 1}}] 2022-03-08T23:24:25.505 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:24:26.111 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:24:26.112 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:24:26.012190Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:24:26.012190Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:24:26.012190Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:24:26.012190Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:26.012190Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:26.012190Z", "running": 0, "size": 1}}] 2022-03-08T23:24:36.132 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:24:37.601 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:24:37.601 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:24:37.492842Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:24:37.492842Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:24:37.492842Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:24:37.492842Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:37.492842Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:37.492842Z", "running": 0, "size": 1}}] 2022-03-08T23:24:47.624 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:24:50.751 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:24:50.751 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:24:50.604348Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:24:50.604348Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:24:50.604348Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:24:50.604348Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:50.604348Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:50.604348Z", "running": 0, "size": 1}}] 2022-03-08T23:25:00.776 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:25:01.408 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:25:01.408 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:25:01.308783Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:25:01.308783Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:25:01.308783Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:25:01.308783Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:01.308783Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:01.308783Z", "running": 0, "size": 1}}] 2022-03-08T23:25:11.429 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:25:12.026 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:25:12.026 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:25:11.933013Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:25:11.933013Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:25:11.933013Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:25:11.933013Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:11.933013Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:11.933013Z", "running": 0, "size": 1}}] 2022-03-08T23:25:22.053 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:25:22.695 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:25:22.695 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:25:22.581401Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:25:22.581401Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:25:22.581401Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:25:22.581401Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:22.581401Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:22.581401Z", "running": 0, "size": 1}}] 2022-03-08T23:25:32.718 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:25:33.355 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:25:33.356 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:25:33.246307Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:25:33.246307Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:25:33.246307Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:25:33.246307Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:33.246307Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:33.246307Z", "running": 0, "size": 1}}] 2022-03-08T23:25:43.376 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:25:44.010 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:25:44.011 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:25:43.914796Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:25:43.914796Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:25:43.914796Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:25:43.914796Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:43.914796Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:43.914796Z", "running": 0, "size": 1}}] 2022-03-08T23:25:54.050 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:25:54.649 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:25:54.650 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:25:54.555365Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:25:54.555365Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:25:54.555365Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:25:54.555365Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:54.555365Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:54.555365Z", "running": 0, "size": 1}}] 2022-03-08T23:26:04.671 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:26:05.238 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:26:05.238 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:26:05.149989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:26:05.149989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:26:05.149989Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:26:05.149989Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:05.149989Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:05.149989Z", "running": 0, "size": 1}}] 2022-03-08T23:26:15.259 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:26:15.863 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:26:15.863 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:26:15.767791Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:26:15.767791Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:26:15.767791Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:26:15.767791Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:15.767791Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:15.767791Z", "running": 0, "size": 1}}] 2022-03-08T23:26:25.888 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:26:26.566 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:26:26.566 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:26:26.470827Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:26:26.470827Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:26:26.470827Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:26:26.470827Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:26.470827Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:26.470827Z", "running": 0, "size": 1}}] 2022-03-08T23:26:36.588 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:26:37.220 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:26:37.220 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:26:37.126326Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:26:37.126326Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:26:37.126326Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:26:37.126326Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:37.126326Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:37.126326Z", "running": 0, "size": 1}}] 2022-03-08T23:26:47.246 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:26:47.868 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:26:47.868 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:26:47.778375Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:26:47.778375Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:26:47.778375Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:26:47.778375Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:47.778375Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:47.778375Z", "running": 0, "size": 1}}] 2022-03-08T23:26:57.889 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:26:58.503 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:26:58.503 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:26:58.402635Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:26:58.402635Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:26:58.402635Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:26:58.402635Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:58.402635Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:58.402635Z", "running": 0, "size": 1}}] 2022-03-08T23:27:08.528 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:27:09.153 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:27:09.153 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:27:09.059402Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:27:09.059402Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:27:09.059402Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:27:09.059402Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:09.059402Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:09.059402Z", "running": 0, "size": 1}}] 2022-03-08T23:27:19.176 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:27:19.789 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:27:19.789 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:27:19.625669Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:27:19.625669Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:27:19.625669Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:27:19.625669Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:19.625669Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:19.625669Z", "running": 0, "size": 1}}] 2022-03-08T23:27:29.814 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:27:30.549 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:27:30.550 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:27:30.399335Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:27:30.399335Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:27:30.399335Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:27:30.399335Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:30.399335Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:30.399335Z", "running": 0, "size": 1}}] 2022-03-08T23:27:40.576 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:27:41.187 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:27:41.187 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:27:41.091222Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:27:41.091222Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:27:41.091222Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:27:41.091222Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:41.091222Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:41.091222Z", "running": 0, "size": 1}}] 2022-03-08T23:27:51.215 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:27:51.817 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:27:51.818 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:27:51.721873Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:27:51.721873Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:27:51.721873Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:27:51.721873Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:51.721873Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:51.721873Z", "running": 0, "size": 1}}] 2022-03-08T23:28:01.842 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:28:02.456 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:28:02.456 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:28:02.358694Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:28:02.358694Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:28:02.358694Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:28:02.358694Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:02.358694Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:28:02.358694Z", "running": 0, "size": 1}}] 2022-03-08T23:28:12.481 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:28:13.219 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:28:13.219 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:28:12.994326Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:28:12.994326Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:28:12.994326Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:28:12.994326Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:12.994326Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:28:12.994326Z", "running": 0, "size": 1}}] 2022-03-08T23:28:23.245 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-9xrgh -- ceph orch ls -f json 2022-03-08T23:28:23.876 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-08T23:28:23.877 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:45.000000Z", "last_refresh": "2022-03-08T23:28:23.755150Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:23.000000Z", "last_refresh": "2022-03-08T23:28:23.755150Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:57.000000Z", "last_refresh": "2022-03-08T23:28:23.755150Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:57.000000Z", "last_refresh": "2022-03-08T23:28:23.755150Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:23.755150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:28:23.755150Z", "running": 0, "size": 1}}] 2022-03-08T23:28:23.900 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:23.901 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:23.902 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T23:28:23.903 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:28:23.916 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:23.917 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:23.974 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T23:28:23.987 DEBUG:teuthology.orchestra.run.smithi186:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T23:28:24.091 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T23:28:24.101 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:24.102 DEBUG:teuthology.orchestra.run.smithi186:> kubectl delete -f cluster.yaml 2022-03-08T23:28:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T23:28:24.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:24.257005 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T23:28:24.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:24.565323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:24.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:24.576308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:24.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:24.576335 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:34.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:34.798863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:34.811 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:34.811950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:45.071 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:45.071647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:45.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:45.081811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:55.304 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:55.303967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:55.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:28:55.317114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:05.536 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:05.536138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:05.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:05.548368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:15.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:15.768549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:15.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:15.780613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:26.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:26.006865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:26.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:26.017907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:36.330 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:36.330478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:36.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:36.342334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:46.559 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:46.558751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:46.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:46.571624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:56.788 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:56.788455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:56.798 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:29:56.798924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:07.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:07.108742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:07.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:07.123332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:17.341 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:17.340922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:17.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:17.354593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:27.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:27.605423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:27.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:27.614660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:37.833 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:37.833311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:37.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:37.843259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:48.062 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:48.062146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:48.072 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:48.073260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:58.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:58.295789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:58.308 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:30:58.309354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:08.530 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:08.529907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:08.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:08.542314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:18.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:18.763657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:18.776 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:18.776867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:28.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:28.997516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:29.008 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:29.008835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:39.289 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:39.289635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:39.301 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:39.302566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:49.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:49.519668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:49.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:49.535862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:59.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:59.794119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:59.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:31:59.808060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:10.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:10.029044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:10.041 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:10.042111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:20.261 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:20.261245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:20.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:20.272993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:30.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:30.493606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:30.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:30.504587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:40.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:40.862436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:40.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:40.872184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:51.090 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:51.091147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:51.102 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:32:51.102554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:01.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:01.325335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:01.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:01.337759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:11.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:11.557750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:11.569 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:11.570475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:21.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:21.792631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:21.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:21.804989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:32.025 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:32.026071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:32.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:32.039153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:42.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:42.263549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:42.274 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:42.275718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:52.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:52.499353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:52.511 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:33:52.511900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:02.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:02.733077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:02.744 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:02.744792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:12.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:12.963051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:12.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:12.975009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:23.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:23.190893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:23.240 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:23.240826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:33.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:33.461235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:33.473 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:33.473800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:43.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:43.691398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:43.705 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:43.705589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:53.975 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:53.975950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:53.987 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:34:53.987933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:04.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:04.224650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:04.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:04.233150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:14.452 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:14.453361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:14.466 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:14.467257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:24.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:24.704882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:24.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:24.714980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:24.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:24.799049 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:35:24.790632662 +0000 UTC m=+2506.244910213 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:35:34.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:34.961565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:34.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:34.978566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:45.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:45.239032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:45.252 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:45.253097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:55.469 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:55.469815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:55.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:35:55.484544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:05.723 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:05.723837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:05.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:05.737863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:15.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:15.959375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:15.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:15.972493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:26.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:26.191050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:26.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:26.259367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:36.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:36.479132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:36.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:36.490871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:46.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:46.708073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:46.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:46.716237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:56.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:56.934946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:56.941 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:36:56.942721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:07.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:07.158758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:07.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:07.168646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:17.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:17.390687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:17.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:17.403611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:27.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:27.628350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:27.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:27.640395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:37.860 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:37.861235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:37.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:37.872010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:48.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:48.093890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:48.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:48.106020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:58.365 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:58.366208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:58.378 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:37:58.379477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:08.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:08.625809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:08.636 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:08.637548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:18.886 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:18.887315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:18.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:18.900701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:29.119 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:29.120484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:29.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:29.135402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:39.352 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:39.353713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:39.366 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:39.367565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:49.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:49.588226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:49.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:49.602260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:59.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:59.823553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:59.839 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:38:59.840760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:10.061 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:10.062745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:10.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:10.077124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:20.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:20.296851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:20.308 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:20.309681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:30.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:30.533544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:30.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:30.542879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:40.764 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:40.765480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:40.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:40.779260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:51.003 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:51.003825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:51.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:39:51.017445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:01.237 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:01.238467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:01.250 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:01.252122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:09.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:09.526615 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:40:10.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:10.156709 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:40:10.146+0000 7fc6897e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:40:10.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:40:10.146+0000 7fc6897e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:40:10.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:40:10.146+0000 7fc6897e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:40:10.157 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:40:10.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:10.156735 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:40:10.146+0000 7fc6897e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:40:10.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:40:10.146+0000 7fc6897e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:40:10.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:40:10.146+0000 7fc6897e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:40:10.158 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:40:11.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:11.475828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:11.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:11.490184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:21.705 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:21.706588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:21.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:21.718681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:32.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:32.034259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:32.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:32.047663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:42.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:42.270282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:42.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:42.282190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:52.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:52.541577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:52.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:40:52.554449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:02.775 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:02.775969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:02.788 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:02.789772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:13.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:13.008666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:13.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:13.021614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:23.266 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:23.267505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:23.276 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:23.278148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:33.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:33.498915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:33.509 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:33.511222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:43.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:43.733633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:43.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:43.746408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:53.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:53.964668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:53.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:41:53.976051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:04.194 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:04.195477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:04.207 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:04.207756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:14.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:14.428534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:14.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:14.440626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:24.668 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:24.669189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:24.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:24.678195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:34.894 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:34.895676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:34.906 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:34.907647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:45.140 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:45.141065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:45.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:45.154617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:55.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:55.442602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:55.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:42:55.455516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:05.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:05.675697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:05.688 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:05.689380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:15.908 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:15.909843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:15.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:15.923866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:26.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:26.145942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:26.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:26.157472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:36.375 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:36.376829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:36.389 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:36.390979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:46.631 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:46.631987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:46.640 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:46.641916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:56.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:56.863605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:56.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:43:56.871725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:07.091 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:07.092300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:07.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:07.104608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:17.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:17.319038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:17.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:17.330392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:27.545 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:27.546231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:27.555 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:27.557124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:37.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:37.816581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:37.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:37.828572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:48.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:48.050684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:48.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:48.062100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:58.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:58.282989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:58.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:44:58.293620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:08.527 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:08.528241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:08.537 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:08.537771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:18.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:18.758067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:18.769 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:18.770371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:28.990 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:28.991442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:29.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:29.005371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:39.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:39.227173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:39.240 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:39.240370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:49.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:49.461187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:49.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:49.473986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:59.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:59.694299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:59.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:45:59.707631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:09.921 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:09.922623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:09.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:09.936065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:20.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:20.156100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:20.166 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:20.167923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:30.386 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:30.387437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:30.400 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:30.401649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:40.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:40.619595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:40.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:40.631599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:50.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:50.850000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:50.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:46:50.862585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:01.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:01.081405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:01.096 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:01.097390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:11.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:11.317825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:11.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:11.330477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:21.554 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:21.554937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:21.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:21.568402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:31.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:31.788784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:31.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:31.797941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:42.019 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:42.020380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:42.030 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:42.031838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:52.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:52.252299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:52.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:47:52.264741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:02.481 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:02.482437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:02.495 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:02.495894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:12.776 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:12.777128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:12.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:12.791046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:23.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:23.032136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:23.042 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:23.043644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:33.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:33.281656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:33.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:33.297746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:43.516 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:43.517382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:43.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:43.525844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:53.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:53.745701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:53.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:48:53.759993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:03.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:03.979088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:03.989 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:03.990617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:14.213 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:14.214251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:14.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:14.225890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:24.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:24.446965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:24.457 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:24.458504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:34.684 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:34.684978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:34.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:34.699121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:44.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:44.918167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:44.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:44.929964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:55.151 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:55.152297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:55.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:49:55.168130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:05.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:05.391272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:05.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:05.403603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:15.617 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:15.618003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:15.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:15.631413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:25.864 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:25.864870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:25.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:25.876884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:36.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:36.095114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:36.115 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:36.115856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:46.332 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:46.333439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:46.346 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:46.347144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:56.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:56.581658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:56.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:50:56.595682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:06.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:06.813937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:06.824 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:06.825544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:17.042 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:17.043051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:17.056 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:17.056905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:27.273 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:27.273996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:27.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:27.285240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:37.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:37.504962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:37.514 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:37.515753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:47.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:47.736779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:47.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:47.745450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:57.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:57.969560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:57.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:51:57.977496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:08.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:08.209298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:08.220 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:08.221668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:18.616 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:18.617352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:18.629 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:18.629830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:28.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:28.847529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:28.895 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:28.895917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:39.213 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:39.214427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:39.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:39.224970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:49.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:49.438985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:49.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:49.450702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:59.673 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:59.674188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:59.684 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:52:59.686040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:09.906 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:09.906868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:09.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:09.919064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:20.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:20.144156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:20.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:20.156938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:30.376 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:30.376783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:30.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:30.413762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:40.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:40.653101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:40.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:40.662920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:40.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:40.975261 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:53:40.964808035 +0000 UTC m=+3602.419085572 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:53:50.909 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:50.910435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:50.921 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:53:50.922537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:01.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:01.140008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:01.152 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:01.153351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:11.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:11.375022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:11.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:11.386675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:21.599 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:21.600828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:21.611 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:21.612424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:31.833 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:31.834692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:31.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:31.847620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:42.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:42.166063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:42.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:42.178571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:52.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:52.433525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:52.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:54:52.445461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:02.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:02.666146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:02.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:02.679745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:12.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:12.900489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:12.911 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:12.912412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:23.128 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:23.129215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:23.140 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:23.141530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:33.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:33.355556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:33.363 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:33.364145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:43.588 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:43.588946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:43.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:43.598669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:53.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:53.816429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:53.828 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:55:53.829321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:04.073 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:04.074454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:04.090 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:04.086867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:14.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:14.304366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:14.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:14.317513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:24.554 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:24.555832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:24.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:24.597880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:34.825 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:34.826831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:34.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:34.839161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:45.057 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:45.058576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:45.069 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:45.070767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:50.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:50.166551 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-08T23:56:50.790 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:50.791670 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:56:50.779+0000 7f39c00fa340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:56:50.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:56:50.779+0000 7f39c00fa340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:56:50.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:56:50.779+0000 7f39c00fa340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:56:50.795 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:56:50.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:50.791710 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:56:50.779+0000 7f39c00fa340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:56:50.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:56:50.779+0000 7f39c00fa340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:56:50.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08T23:56:50.779+0000 7f39c00fa340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:56:50.796 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:56:55.290 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:55.291908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:55.339 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:56:55.340732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:05.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:05.563527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:05.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:05.575437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:15.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:15.795526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:15.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:15.808416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:26.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:26.050699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:26.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:26.066022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:36.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:36.292184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:36.304 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:36.305678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:46.550 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:46.551813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:46.560 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:46.561316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:56.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:56.778832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:56.790 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:57:56.791749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:07.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:07.012630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:07.024 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:07.026113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:17.246 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:17.247493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:17.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:17.260130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:27.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:27.481038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:27.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:27.493321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:37.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:37.707594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:37.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:37.718955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:47.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:47.934745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:47.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:47.946764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:58.186 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:58.188097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:58.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:58:58.199186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:08.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:08.415666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:08.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:08.428771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:18.649 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:18.650127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:18.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:18.662935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:28.903 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:28.904341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:28.916 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:28.917651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:39.134 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:39.136078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:39.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:39.154863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:49.373 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:49.373959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:49.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:49.387001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:59.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:59.602724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:59.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-08 23:59:59.615012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:09.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:09.915163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:09.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:09.930501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:20.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:20.169902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:20.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:20.178512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:30.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:30.621725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:30.631 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:30.633657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:40.852 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:40.853824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:40.865 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:40.867029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:51.083 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:51.084864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:51.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:00:51.096518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:01.314 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:01.314982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:01.326 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:01.327869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:11.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:11.576407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:11.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:11.589056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:21.969 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:21.970355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:21.982 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:21.981858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:32.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:32.204437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:32.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:32.215567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:42.435 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:42.436837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:42.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:42.447857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:52.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:52.675402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:52.686 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:01:52.687666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:02.909 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:02.910408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:02.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:02.923733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:13.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:13.170466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:13.182 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:13.183885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:23.434 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:23.435306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:23.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:23.449002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:33.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:33.675485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:33.727 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:33.728922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:43.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:43.951510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:43.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:43.964133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:54.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:54.180320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:54.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:02:54.193086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:04.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:04.412987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:04.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:04.424822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:14.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:14.678652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:14.688 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:14.688230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:24.916 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:24.917629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:24.924 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:24.925778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:35.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:35.166605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:35.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:35.181377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:45.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:45.402457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:45.417 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:45.418742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:55.636 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:55.637132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:55.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:03:55.648882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:05.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:05.868075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:05.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:05.880170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:16.096 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:16.096837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:16.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:16.105570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:26.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:26.323411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:26.334 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:26.335860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:36.558 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:36.559523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:36.570 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:36.571623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:46.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:46.796663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:46.820 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:46.814345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:57.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:57.035307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:57.086 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:04:57.087921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:07.326 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:07.327771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:07.341 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:07.342578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:17.569 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:17.570340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:17.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:17.579590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:27.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:27.795556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:27.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:27.808426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:38.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:38.109353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:38.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:38.120931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:48.352 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:48.352924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:48.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:48.367814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:58.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:58.619938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:58.632 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:05:58.633565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:08.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:08.876813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:08.884 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:08.886066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:19.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:19.176701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:19.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:19.185413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:29.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:29.401935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:29.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:29.410874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:39.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:39.627837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:39.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:39.638376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:49.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:49.858315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:49.868 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:06:49.869899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:00.087 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:00.088138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:00.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:00.105677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:10.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:10.328916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:10.340 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:10.341548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:20.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:20.575163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:20.609 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:20.609432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:30.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:30.932051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:30.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:30.944667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:41.218 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:41.219159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:41.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:41.230067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:51.469 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:51.470208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:51.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:07:51.483996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:01.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:01.900315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:01.907 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:01.908610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:12.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:12.210766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:12.218 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:12.219322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:22.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:22.439033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:22.452 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:22.452941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:32.673 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:32.673185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:32.684 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:32.684839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:42.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:42.905332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:42.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:42.918003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:53.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:53.142637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:53.151 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:08:53.152508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:03.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:03.369644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:03.378 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:03.378810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:13.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:13.835177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:13.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:13.847958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:24.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:24.066032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:24.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:24.077991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:34.381 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:34.380881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:34.392 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:34.393547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:44.615 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:44.615479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:44.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:44.627967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:54.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:54.849429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:54.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:54.927028 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:09:54.8641186 +0000 UTC m=+4576.318396108 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:09:54.930 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:09:54.928275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:05.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:05.144410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:05.152 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:05.153085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:15.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:15.372795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:15.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:15.388601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:25.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:25.621858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:25.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:25.634719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:35.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:35.858533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:35.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:35.867059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:46.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:46.236832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:46.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:46.247846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:56.468 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:56.467856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:56.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:10:56.480909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:06.750 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:06.751046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:06.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:06.762840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:16.999 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:16.999266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:17.010 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:17.011400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:27.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:27.229519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:27.237 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:27.237897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:37.452 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:37.453014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:37.464 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:37.465607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:47.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:47.697658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:47.710 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:47.710206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:57.924 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:57.924066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:57.938 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:11:57.937891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:08.195 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:08.195653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:08.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:08.208326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:18.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:18.478763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:18.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:18.491565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:28.711 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:28.711178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:28.722 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:28.723071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:38.976 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:38.976540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:38.989 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:38.989831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:49.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:49.203223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:49.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:49.215867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:59.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:59.431393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:06.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:12:59.442154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:09.657 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:09.657282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:09.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:09.669434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:19.889 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:19.889156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:19.904 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:19.904920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:30.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:30.123934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:30.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:30.133847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:30.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:30.797701 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T00:13:31.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:31.439566 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:13:31.426+0000 7f5552a6a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:13:31.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:13:31.426+0000 7f5552a6a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:13:31.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:13:31.426+0000 7f5552a6a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:13:31.441 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:13:31.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:31.439591 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:13:31.426+0000 7f5552a6a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:13:31.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:13:31.426+0000 7f5552a6a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:13:31.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:13:31.426+0000 7f5552a6a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:13:31.442 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:13:40.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:40.414373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:40.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:40.427429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:50.791 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:50.790951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:50.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:13:50.804076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:01.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:01.020764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:01.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:01.032036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:11.254 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:11.254570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:11.266 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:11.267284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:21.484 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:21.484000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:21.496 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:21.496770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:31.722 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:31.723000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:31.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:31.735180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:41.957 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:41.957056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:41.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:41.969070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:52.350 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:52.350363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:52.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:14:52.358153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:02.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:02.575921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:03.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:02.589019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:12.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:12.872456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:12.884 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:12.884776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:23.116 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:23.116333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:23.176 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:23.176977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:33.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:33.448468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:33.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:33.463258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:43.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:43.683122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:43.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:43.696196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:54.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:54.016287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:54.025 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:15:54.025829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:04.246 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:04.246298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:04.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:04.258979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:14.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:14.479703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:14.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:14.493516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:24.713 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:24.713557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:24.724 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:24.725243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:34.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:34.950059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:34.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:34.963111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:45.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:45.191316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:45.202 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:45.202772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:55.452 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:55.452559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:55.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:16:55.463230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:05.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:05.678635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:05.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:05.691250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:15.915 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:15.915767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:15.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:15.932002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:26.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:26.282071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:26.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:26.293313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:36.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:36.534275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:36.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:36.546896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:46.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:46.787242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:46.796 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:46.797071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:57.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:57.066251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:57.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:17:57.076968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:07.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:07.321554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:07.334 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:07.334483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:17.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:17.561921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:17.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:17.575398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:27.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:27.984901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:27.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:27.996518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:38.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:38.215044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:38.266 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:38.266700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:48.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:48.489442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:48.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:48.501492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:58.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:58.720110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:00.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:18:58.732398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:08.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:08.955836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:08.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:08.967586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:19.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:19.187725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:19.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:19.198151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:29.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:29.413173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:31.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:29.424462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:39.646 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:39.646406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:39.657 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:39.657465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:49.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:49.882356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:49.915 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:19:49.915641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:00.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:00.144321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:00.163 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:00.154814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:10.377 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:10.377454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:10.389 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:10.389839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:20.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:20.612630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:20.622 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:20.623403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:30.843 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:30.843417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:30.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:30.855694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:41.079 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:41.079527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:41.087 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:41.088481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:51.309 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:51.309716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:51.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:20:51.322392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:01.545 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:01.545700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:01.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:01.556227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:11.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:11.777449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:11.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:11.790106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:22.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:22.011675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:22.024 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:22.024635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:32.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:32.277503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:32.392 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:32.289871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:42.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:42.507722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:42.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:42.521018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:52.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:52.739778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:52.750 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:21:52.750205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:02.970 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:02.970104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:03.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:02.983370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:13.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:13.209884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:13.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:13.222738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:23.443 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:23.443471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:23.617 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:23.516004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:33.741 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:33.741169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:33.748 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:33.749511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:43.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:43.971658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:43.990 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:43.985898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:54.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:54.205022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:54.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:22:54.217860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:04.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:04.438247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:04.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:04.453251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:14.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:14.678080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:14.691 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:14.692213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:24.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:24.913961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:25.146 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:24.927523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:35.159 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:35.159422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:35.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:35.168234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:45.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:45.391078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:45.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:45.404627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:55.628 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:55.629088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:55.640 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:23:55.640719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:05.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:05.871191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:05.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:05.884262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:16.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:16.107382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:16.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:16.120466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:26.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:26.357775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:26.370 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:26.370840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:36.611 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:36.611625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:36.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:36.624273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:46.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:46.845639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:46.856 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:46.857432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:57.083 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:57.084030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:57.096 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:24:57.095607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:07.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:07.317622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:07.641 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:07.329882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:17.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:17.575564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:17.584 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:17.585378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:27.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:27.870572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:27.877 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:27.877939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:38.098 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:38.098885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:38.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:38.111498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:48.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:48.345717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:48.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:48.355760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:58.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:58.576079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:58.609 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:25:58.610942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:08.828 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:08.829315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:09.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:08.840892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:19.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:19.088557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:19.098 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:19.098700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:29.315 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:29.315498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:29.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:29.330588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:39.547 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:39.548263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:39.668 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:39.557437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:49.775 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:49.774899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:49.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:26:49.784213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:00.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:00.004923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:00.012 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:00.013469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:10.181 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:10.182326 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:27:10.172873039 +0000 UTC m=+5611.627150603 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:27:10.362 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:10.230709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:10.362 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:10.243952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:20.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:20.465822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:20.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:20.477937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:30.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:30.699762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:30.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:30.715279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:40.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:40.936418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:41.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:40.968294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:51.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:51.190113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:51.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:27:51.205194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:01.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:01.425483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:01.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:01.437894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:11.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:11.660426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:11.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:11.672414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:21.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:21.896578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:21.909 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:21.910530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:32.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:32.130765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:32.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:32.145660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:32.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:32.145691 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:42.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:42.372702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:42.384 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:42.385117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:52.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:52.611119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:52.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:28:52.623017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:02.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:02.847351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:02.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:02.861889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:13.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:13.090218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:13.102 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:13.103321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:23.323 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:23.323345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:23.434 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:23.338967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:33.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:33.557763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:33.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:33.567748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:43.784 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:43.785637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:43.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:43.799999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:54.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:54.027143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:54.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:29:54.037835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:04.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:04.256093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:04.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:04.269661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:11.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:11.449894 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T00:30:12.044 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:12.045378 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:30:12.035+0000 7f1092c10340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:30:12.045 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:30:12.035+0000 7f1092c10340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:30:12.045 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:30:12.035+0000 7f1092c10340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:30:12.045 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:30:12.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:12.045402 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:30:12.035+0000 7f1092c10340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:30:12.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:30:12.035+0000 7f1092c10340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:30:12.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:30:12.035+0000 7f1092c10340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:30:12.047 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:30:14.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:14.490493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:14.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:14.500678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:24.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:24.717028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:24.901 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:24.795642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:35.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:35.028820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:35.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:35.038727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:45.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:45.257343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:45.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:45.269630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:55.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:55.494618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:55.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:30:55.507420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:05.761 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:05.762392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:05.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:05.774554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:15.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:15.994708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:16.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:16.006155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:26.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:26.225245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:26.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:26.236465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:36.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:36.456031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:36.518 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:36.467960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:46.691 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:46.691808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:46.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:46.705320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:56.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:56.929069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:56.939 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:31:56.940651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:07.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:07.163068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:07.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:07.210700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:17.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:17.433366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:17.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:17.446013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:27.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:27.669821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:27.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:27.679396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:37.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:37.911458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:37.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:37.924600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:48.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:48.149241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:48.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:48.161172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:58.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:58.384852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:58.588 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:32:58.396769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:08.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:08.613534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:08.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:08.626703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:18.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:18.884293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:18.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:18.897780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:29.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:29.118027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:29.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:29.189495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:39.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:39.403729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:39.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:39.481104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:49.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:49.697008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:49.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:49.708040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:59.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:33:59.966030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:00.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:00.013137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:10.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:10.232866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:10.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:10.264315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:20.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:20.495829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:20.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:20.506989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:30.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:30.721743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:30.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:30.734592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:40.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:40.956892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:40.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:40.968955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:51.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:51.185292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:51.194 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:34:51.195435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:01.416 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:01.417731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:01.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:01.429360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:11.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:11.646643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:11.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:11.660627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:21.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:21.882327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:21.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:21.894279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:32.113 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:32.114315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:32.126 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:32.127479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:42.346 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:42.347422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:42.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:42.359248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:52.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:52.576512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:52.631 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:35:52.588769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:02.806 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:02.807362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:02.816 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:02.818101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:13.036 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:13.037835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:13.048 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:13.050200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:23.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:23.270610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:23.314 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:23.284739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:33.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:33.504962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:33.517 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:33.518261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:43.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:43.740072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:43.750 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:43.752405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:53.969 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:53.970943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:54.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:36:53.984298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:04.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:04.206988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:04.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:04.223785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:14.442 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:14.443508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:14.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:14.455584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:24.675 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:24.676660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:24.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:24.692087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:34.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:34.911954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:34.923 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:34.925378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:45.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:45.175398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:45.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:45.185021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:55.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:55.415193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:55.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:37:55.428087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:05.646 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:05.647556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:05.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:05.691917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:15.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:15.906645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:15.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:15.919364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:26.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:26.163499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:26.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:26.176210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:36.398 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:36.399318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:36.411 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:36.412653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:46.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:46.631320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:46.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:46.646728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:56.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:56.868520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:56.877 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:38:56.879322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:07.096 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:07.097480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:07.106 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:07.107683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:17.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:17.329178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:17.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:17.343460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:27.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:27.569017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:27.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:27.582224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:37.801 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:37.803043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:37.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:37.814500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:48.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:48.035755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:48.047 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:48.048805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:58.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:58.270318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:58.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:39:58.283736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:08.555 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:08.556431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:08.585 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:08.586302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:18.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:18.805269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:18.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:18.862930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:29.083 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:29.084674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:29.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:29.094720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:39.313 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:39.315083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:39.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:39.323065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:49.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:49.542670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:49.555 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:49.557244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:59.776 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:59.777345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:59.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:40:59.788708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:10.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:10.008637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:10.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:10.020971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:20.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:20.242195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:20.249 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:20.251056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:30.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:30.472446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:31.024 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:30.486304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:40.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:40.705985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:40.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:40.717301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:50.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:50.934335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:53.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:41:50.947103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:01.166 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:01.168134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:01.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:01.178667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:11.451 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:11.453033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:11.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:11.461961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:21.684 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:21.685839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:21.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:21.697063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:31.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:31.914178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:31.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:31.923550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:42.140 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:42.141346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:43.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:42.151552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:52.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:52.373368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:52.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:42:52.387406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:02.609 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:02.610651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:02.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:02.622048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:12.839 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:12.840937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:20.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:12.859899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:23.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:23.082060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:23.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:23.093384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:33.313 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:33.314462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:33.327 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:33.328846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:43.554 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:43.555748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:45.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:43.566491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:53.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:53.804460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:53.816 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:43:53.818289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:04.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:04.035297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:07.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:04.044080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:14.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:14.258509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:14.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:14.272331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:24.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:24.487425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:32.419 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:24.499993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:34.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:34.721017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:34.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:34.732699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:44.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:44.996917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:45.010 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:45.011770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:55.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:55.295915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:55.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:44:55.308661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:05.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:05.525489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:11.592 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:05.537121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:15.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:15.763155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:15.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:15.774874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:25.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:25.919305 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:45:25.909635754 +0000 UTC m=+6707.363913312 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:45:25.991 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:25.992838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:26.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:26.005350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:36.228 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:36.229696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:36.239 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:36.240500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:46.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:46.462167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:46.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:46.479350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:56.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:56.695471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:56.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:45:56.705422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:06.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:06.951785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:06.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:06.959767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:17.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:17.206479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:17.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:17.216585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:27.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:27.434858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:27.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:27.447232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:37.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:37.663893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:37.673 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:37.675036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:47.932 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:47.934022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:47.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:47.946345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:52.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:52.055801 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T00:46:52.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:52.697853 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:46:52.688+0000 7fdf181b1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:46:52.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:46:52.688+0000 7fdf181b1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:46:52.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:46:52.688+0000 7fdf181b1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:46:52.697 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:46:52.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:52.697874 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:46:52.688+0000 7fdf181b1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:46:52.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:46:52.688+0000 7fdf181b1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:46:52.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T00:46:52.688+0000 7fdf181b1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:46:52.698 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:46:58.201 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:58.202285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:58.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:46:58.211427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:08.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:08.466666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:08.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:08.479423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:18.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:18.698936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:18.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:18.710626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:28.924 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:28.925031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:28.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:28.938422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:39.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:39.159211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:39.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:39.169268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:49.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:49.388944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:49.394 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:49.396491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:59.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:59.620147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:59.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:47:59.629417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:09.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:09.851298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:09.860 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:09.862296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:20.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:20.089536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:20.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:20.102543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:30.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:30.329465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:30.340 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:30.341994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:40.564 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:40.565332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:40.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:40.579416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:50.801 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:50.802457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:50.814 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:48:50.815874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:01.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:01.036245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:01.048 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:01.050075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:11.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:11.270407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:11.323 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:11.324451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:21.542 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:21.543284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:21.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:21.554763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:31.779 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:31.780666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:31.790 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:31.792531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:42.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:42.017282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:42.030 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:42.031681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:52.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:52.252079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:52.264 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:49:52.265532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:02.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:02.482086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:02.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:02.492934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:12.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:12.707628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:12.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:12.717744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:22.937 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:22.939188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:22.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:22.954704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:33.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:33.175729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:33.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:33.187483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:43.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:43.434125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:43.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:43.445965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:53.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:53.666513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:53.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:50:53.678319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:03.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:03.898511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:03.908 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:03.910544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:14.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:14.131873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:14.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:14.141030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:24.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:24.360446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:24.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:24.372642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:34.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:34.591038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:34.603 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:34.604717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:44.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:44.835650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:44.852 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:44.853721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:55.071 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:55.072414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:55.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:51:55.085443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:05.300 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:05.301802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:05.314 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:05.315948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:15.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:15.536376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:15.544 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:15.545551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:25.760 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:25.762157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:25.771 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:25.773171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:35.991 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:35.993341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:36.002 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:36.003672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:46.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:46.228458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:46.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:46.240651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:56.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:56.465370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:56.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:52:56.479050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:06.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:06.697417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:06.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:06.708827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:16.930 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:16.931767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:16.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:16.942759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:27.159 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:27.160409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:27.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:27.172084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:37.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:37.391876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:37.405 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:37.407377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:47.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:47.629190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:47.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:47.640967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:57.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:57.859317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:57.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:53:57.919492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:08.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:08.158629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:08.172 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:08.174393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:18.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:18.392742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:18.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:18.404300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:28.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:28.636788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:28.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:28.647128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:38.915 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:38.916600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:38.961 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:38.962362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:49.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:49.181024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:49.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:49.191332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:59.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:59.409193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:59.418 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:54:59.420010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:09.638 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:09.640030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:09.649 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:09.651298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:19.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:19.873797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:19.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:19.887419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:30.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:30.109374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:30.121 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:30.122824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:40.344 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:40.345619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:40.356 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:40.358563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:50.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:50.578539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:50.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:55:50.592251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:00.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:00.814922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:00.825 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:00.826992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:11.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:11.051351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:11.063 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:11.065226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:21.310 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:21.312050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:21.327 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:21.327892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:31.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:31.551159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:31.561 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:31.563271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:41.786 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:41.787877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:41.798 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:41.799918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:52.024 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:52.023214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:52.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:56:52.036396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:02.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:02.256969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:02.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:02.269583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:12.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:12.490766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:12.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:12.498966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:22.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:22.719055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:22.729 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:22.731380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:32.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:32.956319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:32.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:32.970323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:43.192 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:43.193669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:43.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:43.206015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:53.426 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:53.428124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:53.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:57:53.439031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:03.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:03.662291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:03.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:03.672754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:13.902 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:13.903939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:13.914 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:13.916729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:24.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:24.176860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:24.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:24.187954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:34.416 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:34.417479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:34.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:34.435003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:44.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:44.691265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:44.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:44.700197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:54.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:54.924307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:54.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:58:54.968410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:05.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:05.190758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:05.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:05.202270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:15.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:15.422682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:15.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:15.438325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:25.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:25.661182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:25.672 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:25.674489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:35.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:35.895512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:35.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:35.907144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:46.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:46.126308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:46.138 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:46.140581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:56.353 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:56.355003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:56.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 00:59:56.369889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:06.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:06.589279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:06.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:06.602152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:16.823 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:16.824905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:16.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:16.834656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:27.056 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:27.057884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:27.068 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:27.070458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:37.289 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:37.290448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:37.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:37.301531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:47.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:47.521677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:47.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:47.530827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:57.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:57.755379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:57.764 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:00:57.766073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:07.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:07.988942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:08.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:08.048894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:18.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:18.265004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:18.275 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:18.277139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:28.490 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:28.491617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:28.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:28.505692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:38.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:38.883830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:38.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:38.893743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:49.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:49.114164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:49.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:49.125664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:59.496 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:59.498248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:59.509 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:01:59.511504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:09.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:09.796448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:09.814 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:09.816638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:20.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:20.068261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:20.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:20.082119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:30.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:30.488554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:30.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:30.502949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:40.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:40.849369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:40.859 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:40.861069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:41.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:41.146200 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:02:41.135139694 +0000 UTC m=+7742.589417263 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:02:51.078 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:51.079539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:51.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:02:51.090736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:01.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:01.399301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:01.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:01.411813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:11.629 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:11.631134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:11.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:11.642940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:21.865 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:21.866842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:21.919 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:21.920908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:32.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:32.143915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:32.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:32.157650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:32.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:32.706079 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T01:03:33.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:33.318842 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:03:33.309+0000 7fd6e48c2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:03:33.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:03:33.309+0000 7fd6e48c2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:03:33.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:03:33.309+0000 7fd6e48c2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:03:33.318 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:03:33.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:33.318873 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:03:33.309+0000 7fd6e48c2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:03:33.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:03:33.309+0000 7fd6e48c2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:03:33.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:03:33.309+0000 7fd6e48c2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:03:33.319 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:03:42.383 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:42.385886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:42.394 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:42.397023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:52.613 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:52.614676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:52.626 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:03:52.627824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:02.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:02.847350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:02.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:02.860616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:13.082 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:13.083193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:13.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:13.095528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:23.315 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:23.316930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:23.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:23.330529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:33.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:33.551256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:33.561 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:33.563308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:43.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:43.854824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:43.863 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:43.865664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:54.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:54.172748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:54.181 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:04:54.184117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:04.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:04.405180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:04.418 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:04.420123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:14.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:14.641052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:14.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:14.654128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:24.877 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:24.879078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:24.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:24.892399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:35.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:35.114123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:35.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:35.125155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:45.341 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:45.343512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:45.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:45.355415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:55.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:55.596857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:55.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:05:55.607671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:05.829 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:05.827021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:05.840 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:05.842838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:16.140 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:16.142422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:16.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:16.156229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:26.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:26.371142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:26.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:26.381570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:36.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:36.602183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:36.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:36.613963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:47.019 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:47.020922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:47.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:47.055186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:57.275 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:57.276693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:57.287 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:06:57.288919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:07.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:07.551178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:07.561 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:07.563880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:17.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:17.953154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:17.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:17.964841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:28.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:28.182362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:28.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:28.192338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:38.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:38.454781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:38.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:38.464816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:48.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:48.678735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:48.701 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:48.699543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:58.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:58.948654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:58.957 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:07:58.959653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:09.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:09.183317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:09.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:09.192563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:19.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:19.406244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:19.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:19.416816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:29.638 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:29.640573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:29.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:29.649259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:39.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:39.872113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:39.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:39.886617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:50.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:50.170738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:50.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:08:50.179570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:00.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:00.415342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:00.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:00.429039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:10.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:10.649334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:10.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:10.662557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:21.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:21.131545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:21.196 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:21.198043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:31.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:31.413386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:31.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:31.485580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:41.742 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:41.743670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:41.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:41.756736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:51.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:51.972906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:52.010 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:09:52.011780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:02.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:02.232539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:02.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:02.245066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:12.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:12.464307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:12.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:12.480289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:22.712 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:22.713281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:22.724 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:22.725725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:32.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:32.945538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:32.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:32.960471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:43.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:43.181274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:43.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:43.192264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:53.411 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:53.413029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:53.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:10:53.427516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:03.646 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:03.647599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:03.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:03.657969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:13.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:13.873177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:13.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:13.885645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:24.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:24.105681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:24.116 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:24.117867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:34.336 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:34.337573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:34.349 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:34.350130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:44.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:44.572374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:44.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:44.583309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:54.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:54.799056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:54.810 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:11:54.812249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:05.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:05.027753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:05.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:05.039982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:15.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:15.259341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:15.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:15.270997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:25.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:25.494490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:25.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:25.507661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:35.726 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:35.727494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:35.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:35.739314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:45.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:45.955330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:45.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:45.966608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:56.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:56.184660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:56.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:12:56.198734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:06.464 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:06.466037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:06.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:06.474081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:16.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:16.690306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:16.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:16.700983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:26.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:26.936107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:26.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:26.947695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:37.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:37.163063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:37.176 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:37.175603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:47.424 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:47.425851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:47.445 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:47.446171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:57.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:57.665185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:57.672 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:13:57.673986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:07.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:07.919715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:07.930 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:07.932102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:18.152 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:18.153517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:18.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:18.166092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:28.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:28.403959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:28.415 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:28.416559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:38.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:38.636225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:38.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:38.649144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:48.901 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:48.902190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:48.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:48.937710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:59.249 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:59.251057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:59.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:14:59.260358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:09.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:09.501151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:09.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:09.513604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:19.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:19.732188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:19.742 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:19.743383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:30.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:30.006305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:30.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:30.017538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:40.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:40.233169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:40.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:40.245248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:50.563 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:50.564091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:50.573 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:15:50.574537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:00.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:00.793583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:00.805 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:00.806422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:11.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:11.029010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:11.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:11.038885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:21.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:21.260585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:21.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:21.274172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:31.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:31.493536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:31.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:31.506473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:41.728 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:41.729099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:41.741 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:41.742430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:52.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:52.054461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:52.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:16:52.067607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:02.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:02.289591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:02.300 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:02.301912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:12.517 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:12.518273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:12.529 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:12.530298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:22.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:22.758003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:22.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:22.768412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:32.991 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:32.991943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:33.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:33.005826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:43.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:43.225654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:43.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:43.239725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:53.458 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:53.458824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:53.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:17:53.472607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:03.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:03.722176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:03.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:03.735050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:13.952 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:13.952952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:13.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:13.964977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:24.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:24.180264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:24.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:24.192010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:34.411 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:34.411993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:34.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:34.423879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:44.642 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:44.642756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:44.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:44.652892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:54.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:54.873393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:54.884 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:18:54.885277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:05.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:05.125895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:05.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:05.172087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:15.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:15.430123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:15.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:15.442129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:25.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:25.664550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:25.675 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:25.676725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:35.898 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:35.899254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:35.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:35.913000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:46.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:46.134224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:46.149 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:46.150038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:56.370 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:56.370550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:56.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:56.381067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:56.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:19:56.413973 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:19:56.406069264 +0000 UTC m=+8777.860346813 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:20:06.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:06.608809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:06.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:06.621808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:13.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:13.325836 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T01:20:13.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:13.954617 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:20:13.945+0000 7fc07a533340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:20:13.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:20:13.945+0000 7fc07a533340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:20:13.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:20:13.945+0000 7fc07a533340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:20:13.955 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:20:13.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:13.954646 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:20:13.945+0000 7fc07a533340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:20:13.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:20:13.945+0000 7fc07a533340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:20:13.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:20:13.945+0000 7fc07a533340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:20:13.956 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:20:16.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:16.872451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:16.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:16.883641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:27.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:27.124562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:27.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:27.134393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:37.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:37.354725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:37.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:37.370400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:47.595 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:47.596351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:47.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:47.611284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:57.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:57.828581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:57.838 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:20:57.839302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:08.063 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:08.063732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:08.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:08.148158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:18.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:18.367905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:18.381 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:18.382700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:28.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:28.620390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:28.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:28.637323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:38.865 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:38.865575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:38.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:38.880029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:49.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:49.112595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:49.122 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:49.123146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:59.375 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:59.376010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:59.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:21:59.386728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:09.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:09.604662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:09.615 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:09.615453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:19.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:19.835803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:19.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:19.845256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:30.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:30.064602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:30.079 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:30.079099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:40.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:40.296415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:40.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:40.308619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:50.533 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:50.534194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:50.556 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:22:50.556694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:00.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:00.777994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:00.831 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:00.831874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:11.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:11.052031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:11.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:11.065609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:21.287 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:21.287939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:21.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:21.299438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:31.547 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:31.547662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:31.560 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:31.561298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:41.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:41.788064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:41.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:41.816390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:52.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:52.035867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:52.047 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:23:52.048224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:02.274 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:02.274390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:02.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:02.286868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:12.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:12.532887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:12.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:12.541172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:22.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:22.763066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:22.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:22.773207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:33.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:33.007274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:33.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:33.017830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:43.273 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:43.274206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:43.287 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:43.286673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:53.508 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:53.508456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:53.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:24:53.520512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:03.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:03.738023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:03.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:03.749927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:14.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:14.007607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:14.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:14.018040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:24.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:24.271612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:24.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:24.284761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:34.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:34.503565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:34.513 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:34.514483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:44.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:44.734328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:44.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:44.746031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:54.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:54.966009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:54.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:25:54.979565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:05.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:05.215751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:05.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:05.227444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:15.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:15.448271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:15.459 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:15.460400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:25.680 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:25.679229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:25.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:25.690998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:35.914 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:35.915448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:35.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:35.927441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:46.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:46.147969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:46.161 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:46.160291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:56.378 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:56.379332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:56.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:26:56.391911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:06.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:06.614511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:06.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:06.625851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:16.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:16.844808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:16.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:16.856713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:27.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:27.138283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:27.146 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:27.147340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:37.362 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:37.363340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:37.376 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:37.377562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:47.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:47.595458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:47.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:47.605141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:57.826 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:57.826942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:57.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:27:57.838299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:08.055 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:08.055948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:08.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:08.066485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:18.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:18.281248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:18.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:18.293465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:28.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:28.513251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:28.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:28.522304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:38.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:38.740399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:38.752 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:38.753339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:38.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:38.753386 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:48.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:48.975358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:48.987 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:48.988812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:59.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:59.205275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:59.217 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:28:59.218628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:09.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:09.447613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:09.456 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:09.457040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:19.675 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:19.675883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:19.687 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:19.688279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:29.909 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:29.909554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:29.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:29.919133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:40.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:40.159182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:40.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:40.174395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:50.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:50.391073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:50.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:29:50.403810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:00.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:00.640288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:00.649 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:00.650366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:10.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:10.867202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:10.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:10.881854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:21.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:21.105427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:21.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:21.118487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:31.338 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:31.338551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:31.349 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:31.350819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:41.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:41.568774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:41.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:41.581705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:51.801 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:51.801524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:51.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:30:51.814927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:02.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:02.040492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:02.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:02.053403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:12.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:12.271611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:12.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:12.285238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:22.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:22.541280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:22.552 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:22.553669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:32.776 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:32.776850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:32.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:32.790250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:43.013 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:43.013681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:43.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:43.022127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:53.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:53.247996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:53.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:31:53.260680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:03.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:03.497516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:03.539 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:03.535285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:13.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:13.762637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:13.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:13.774655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:23.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:23.997723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:24.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:24.024636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:34.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:34.293465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:34.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:34.304198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:44.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:44.524226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:44.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:44.536517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:54.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:54.790005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:54.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:32:54.801281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:05.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:05.039647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:05.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:05.050213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:15.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:15.269354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:15.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:15.282366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:25.502 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:25.503435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:25.514 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:25.515249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:35.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:35.733792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:35.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:35.746384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:45.964 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:45.965008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:45.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:45.978142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:56.221 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:56.222276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:56.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:33:56.233277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:06.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:06.454278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:06.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:06.466054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:16.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:16.683462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:16.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:16.695175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:26.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:26.913961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:26.924 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:26.925431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:37.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:37.144137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:37.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:37.154255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:47.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:47.371910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:47.381 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:47.382777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:57.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:57.602131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:57.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:34:57.613645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:07.833 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:07.834284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:07.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:07.846507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:18.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:18.064776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:18.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:18.078132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:28.302 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:28.303085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:28.314 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:28.315512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:38.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:38.536397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:38.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:38.547430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:48.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:48.769105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:48.781 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:48.781945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:59.002 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:59.003175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:59.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:35:59.017373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:09.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:09.237420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:09.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:09.249381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:19.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:19.475605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:19.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:19.488249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:29.713 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:29.714050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:29.723 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:29.724194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:39.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:39.943243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:39.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:39.955559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:50.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:50.175335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:50.186 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:50.187472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:53.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:53.962860 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T01:36:54.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:54.618815 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:36:54.609+0000 7f8f58321340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:36:54.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:36:54.609+0000 7f8f58321340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:36:54.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:36:54.609+0000 7f8f58321340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:36:54.619 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:36:54.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:36:54.618842 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:36:54.609+0000 7f8f58321340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:36:54.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:36:54.609+0000 7f8f58321340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:36:54.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:36:54.609+0000 7f8f58321340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:36:54.620 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:37:00.405 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:00.406012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:00.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:00.421915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:10.663 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:10.663986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:10.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:10.679291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:11.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:11.074762 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:37:11.066265879 +0000 UTC m=+9812.520543434 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:37:20.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:20.911302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:20.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:20.925947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:31.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:31.180528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:31.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:31.191948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:41.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:41.455490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:41.467 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:41.467893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:51.686 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:51.687583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:51.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:37:51.698438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:01.919 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:01.919531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:01.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:01.932961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:12.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:12.158007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:12.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:12.168437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:22.386 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:22.387037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:22.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:22.396370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:32.613 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:32.613706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:32.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:32.626817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:42.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:42.850610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:42.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:42.868663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:53.087 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:53.087830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:53.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:38:53.101595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:03.323 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:03.323801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:03.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:03.338954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:13.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:13.560606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:13.573 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:13.573391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:23.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:23.832954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:23.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:23.843561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:34.063 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:34.064127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:34.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:34.075979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:44.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:44.293577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:44.305 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:44.306878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:54.526 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:54.526749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:54.538 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:39:54.539753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:04.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:04.763212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:04.775 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:04.775973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:14.996 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:14.997338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:15.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:15.010338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:25.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:25.232000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:25.242 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:25.243206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:35.466 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:35.467067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:35.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:35.479033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:45.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:45.700076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:45.711 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:45.712549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:55.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:55.935442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:55.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:40:55.957504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:06.176 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:06.176554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:06.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:06.188767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:16.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:16.408730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:16.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:16.481364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:26.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:26.750105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:26.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:26.760687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:36.975 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:36.976371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:36.988 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:36.989526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:47.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:47.248595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:47.262 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:47.262337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:57.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:57.479861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:57.545 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:41:57.546249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:07.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:07.775368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:07.788 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:07.789908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:18.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:18.007350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:18.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:18.030773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:28.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:28.267425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:28.279 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:28.280594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:38.511 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:38.512507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:38.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:38.524386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:48.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:48.808677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:48.817 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:48.818346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:59.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:59.052623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:59.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:42:59.061486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:09.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:09.285124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:09.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:09.297899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:19.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:19.521500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:19.538 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:19.539199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:29.915 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:29.916256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:29.925 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:29.926637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:40.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:40.170619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:40.182 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:40.183440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:50.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:50.398284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:50.408 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:43:50.409215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:00.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:00.653532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:00.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:00.665459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:10.889 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:10.890352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:10.901 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:10.902170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:21.119 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:21.119709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:21.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:21.132348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:31.435 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:31.436114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:31.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:31.448726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:41.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:41.665912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:41.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:41.678665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:51.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:51.892252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:51.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:44:51.907077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:02.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:02.124599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:02.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:02.137687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:12.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:12.355533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:12.365 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:12.366845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:22.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:22.605434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:22.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:22.621010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:32.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:32.845888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:32.856 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:32.855944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:43.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:43.074902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:43.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:43.091128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:53.306 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:53.307436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:53.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:45:53.320167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:03.599 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:03.600380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:03.613 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:03.614007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:13.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:13.835631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:13.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:13.847737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:24.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:24.066924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:24.078 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:24.079689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:34.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:34.297490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:34.313 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:34.314725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:44.529 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:44.529998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:44.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:44.543091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:54.760 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:54.760872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:54.771 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:46:54.772849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:04.990 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:04.991497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:05.002 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:05.003388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:15.225 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:15.226216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:15.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:15.239695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:25.458 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:25.459488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:25.468 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:25.469405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:35.688 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:35.689726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:35.702 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:35.703236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:45.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:45.922903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:45.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:45.936330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:56.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:56.258212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:56.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:47:56.270913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:06.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:06.635132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:06.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:06.647861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:16.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:16.867140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:16.878 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:16.879329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:27.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:27.210335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:27.219 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:27.220470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:37.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:37.634886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:37.643 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:37.644596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:47.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:47.949395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:47.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:47.962101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:58.216 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:58.217647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:58.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:48:58.226079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:08.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:08.597595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:08.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:08.608577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:18.836 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:18.837616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:18.852 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:18.853432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:29.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:29.144400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:29.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:29.156117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:39.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:39.563273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:39.573 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:39.574704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:49.790 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:49.791583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:49.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:49:49.804346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:00.018 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:00.019665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:00.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:00.077786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:10.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:10.320253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:10.332 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:10.333452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:20.556 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:20.557448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:20.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:20.605290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:30.824 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:30.825782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:30.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:30.835805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:41.228 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:41.226309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:41.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:41.237897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:51.481 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:51.482350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:51.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:50:51.494020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:01.711 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:01.712237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:01.724 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:01.725891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:11.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:11.944948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:11.991 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:11.992158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:22.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:22.209373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:22.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:22.224500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:32.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:32.483441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:32.495 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:32.497228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:42.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:42.756937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:42.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:42.768924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:52.989 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:52.990015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:53.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:51:53.002050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:03.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:03.249082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:03.264 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:03.265731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:13.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:13.589857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:13.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:13.603093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:23.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:23.850516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:23.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:23.862860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:24.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:24.087085 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:52:24.079169464 +0000 UTC m=+10725.533447013 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:52:34.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:34.192582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:34.198 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:34.200458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:44.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:44.558473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:44.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:44.566905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:54.801 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:54.802045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:54.814 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:52:54.814567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:05.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:05.029462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:05.042 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:05.043906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:15.305 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:15.305998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:15.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:15.319082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:25.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:25.535313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:25.547 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:25.549033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:34.623 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:34.624531 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T01:53:35.234 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:35.234663 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:53:35.225+0000 7fa0f5d33340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:53:35.234 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:53:35.225+0000 7fa0f5d33340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:53:35.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:53:35.225+0000 7fa0f5d33340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:53:35.235 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:53:35.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:35.234689 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:53:35.225+0000 7fa0f5d33340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:53:35.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:53:35.225+0000 7fa0f5d33340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:53:35.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T01:53:35.225+0000 7fa0f5d33340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:53:35.236 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:53:35.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:35.769011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:35.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:35.781527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:45.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:45.999437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:46.010 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:46.012203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:56.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:56.236162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:56.246 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:53:56.248014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:06.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:06.466328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:06.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:06.478973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:16.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:16.700399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:16.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:16.715334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:26.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:26.932890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:26.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:26.945436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:37.163 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:37.164123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:37.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:37.175931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:47.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:47.392357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:47.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:47.405192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:57.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:57.626591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:57.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:54:57.638340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:08.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:08.030037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:08.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:08.040186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:18.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:18.256781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:18.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:18.269279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:28.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:28.502167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:28.509 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:28.510783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:38.730 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:38.731291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:38.746 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:38.744306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:48.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:48.964247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:48.975 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:48.976374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:59.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:59.198432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:59.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:55:59.211804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:09.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:09.432590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:09.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:09.442211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:19.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:19.721443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:19.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:19.732816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:30.098 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:30.099916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:30.114 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:30.116058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:40.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:40.360715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:40.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:40.369917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:50.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:50.605335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:50.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:56:50.632114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:00.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:00.973358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:00.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:00.984932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:11.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:11.215986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:11.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:11.227809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:21.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:21.541267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:21.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:21.554483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:31.778 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:31.779147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:31.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:31.790938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:42.008 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:42.009441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:42.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:42.022564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:52.234 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:52.235952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:52.246 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:57:52.247720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:02.466 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:02.467942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:02.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:02.479825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:12.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:12.697338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:12.708 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:12.710089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:23.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:23.032076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:23.042 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:23.044080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:33.265 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:33.266165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:33.278 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:33.279973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:43.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:43.513176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:43.526 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:43.528173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:53.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:53.775906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:53.786 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:58:53.787997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:04.018 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:04.019828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:04.032 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:04.033603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:14.250 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:14.251929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:14.261 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:14.263250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:24.513 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:24.507160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:24.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:24.521429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:34.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:34.764786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:34.776 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:34.777429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:44.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:44.996103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:45.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:45.008277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:55.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:55.227527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:55.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 01:59:55.239064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:05.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:05.484752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:05.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:05.494348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:15.710 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:15.711707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:15.722 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:15.723480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:25.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:25.999841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:26.010 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:26.012112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:36.262 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:36.264116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:36.274 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:36.276422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:46.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:46.499193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:46.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:46.513138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:56.728 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:56.729547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:56.747 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:00:56.749254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:06.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:06.975538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:06.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:06.984122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:17.202 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:17.203217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:17.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:17.216943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:27.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:27.439940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:27.451 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:27.452716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:37.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:37.671079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:37.679 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:37.680836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:47.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:47.901006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:47.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:47.913614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:58.132 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:58.130565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:58.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:01:58.144595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:08.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:08.365698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:08.378 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:08.379323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:18.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:18.600908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:18.611 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:18.613077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:28.839 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:28.840430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:28.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:28.852161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:39.079 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:39.080437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:39.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:39.094433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:49.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:49.312610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:49.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:49.324300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:59.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:59.549787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:59.561 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:02:59.562329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:09.791 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:09.792743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:09.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:09.803305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:20.044 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:20.045238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:20.056 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:20.057408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:30.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:30.431076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:30.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:30.440528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:40.760 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:40.762097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:40.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:40.801734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:51.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:51.134193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:51.146 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:03:51.147658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:01.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:01.413543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:01.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:01.486242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:11.812 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:11.813270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:11.825 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:11.826200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:22.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:22.089055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:22.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:22.126432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:32.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:32.346145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:32.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:32.381919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:42.606 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:42.607894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:42.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:42.618262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:52.836 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:52.837318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:52.851 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:04:52.852440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:03.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:03.125164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:03.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:03.138223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:13.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:13.359019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:13.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:13.370731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:23.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:23.590308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:23.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:23.602098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:33.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:33.947009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:33.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:33.966275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:44.192 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:44.193841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:44.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:44.206511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:54.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:54.422646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:54.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:05:54.435334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:04.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:04.652636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:04.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:04.667560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:14.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:14.882077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:14.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:14.894518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:25.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:25.113488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:25.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:25.126711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:35.343 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:35.344847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:35.358 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:35.360177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:45.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:45.578070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:45.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:45.592126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:55.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:55.810215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:55.819 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:06:55.821388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:06.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:06.041897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:06.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:06.051611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:16.273 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:16.274271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:16.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:16.286295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:26.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:26.508433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:26.515 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:26.516860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:36.754 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:36.755486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:36.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:36.765552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:46.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:46.987394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:46.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:46.997012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:57.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:57.245300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:57.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:07:57.257728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:07.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:07.498490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:07.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:07.508785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:17.722 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:17.723336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:17.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:17.734117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:27.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:27.966186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:27.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:27.978606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:38.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:38.187692 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:08:38.163304101 +0000 UTC m=+11699.617581652 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:08:38.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:38.212253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:38.220 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:38.222368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:48.469 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:48.470377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:48.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:48.481970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:58.703 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:58.704293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:58.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:08:58.719771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:08.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:08.937595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:08.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:08.949931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:19.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:19.171348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:19.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:19.184744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:29.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:29.403456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:29.469 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:29.470930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:39.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:39.689900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:39.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:39.701079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:49.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:49.974680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:49.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:09:49.987307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:00.201 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:00.202504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:00.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:00.217353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:10.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:10.437078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:10.450 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:10.451845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:15.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:15.243200 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T02:10:15.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:15.861774 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:10:15.854+0000 7f7c6da5b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:10:15.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:10:15.854+0000 7f7c6da5b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:10:15.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:10:15.854+0000 7f7c6da5b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:10:15.862 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:10:15.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:15.861794 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:10:15.854+0000 7f7c6da5b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:10:15.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:10:15.854+0000 7f7c6da5b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:10:15.863 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:10:15.854+0000 7f7c6da5b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:10:15.863 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:10:20.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:20.670923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:20.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:20.683462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:30.902 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:30.903340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:30.914 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:30.915445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:41.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:41.134080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:41.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:41.147623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:51.362 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:51.363878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:51.373 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:10:51.375077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:01.592 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:01.593615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:01.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:01.603403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:11.819 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:11.819811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:11.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:11.833466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:22.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:22.075926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:22.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:22.089817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:32.347 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:32.348560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:32.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:32.360523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:42.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:42.578935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:42.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:42.591296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:52.814 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:52.815426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:52.826 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:11:52.827995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:03.044 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:03.045632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:03.058 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:03.060000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:13.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:13.278899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:13.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:13.287236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:23.558 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:23.559223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:23.570 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:23.571583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:33.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:33.790580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:33.838 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:33.840196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:44.059 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:44.060194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:44.069 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:44.071340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:54.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:54.286456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:54.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:12:54.298454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:04.518 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:04.519738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:04.530 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:04.532236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:14.751 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:14.752398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:14.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:14.765200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:24.987 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:24.988013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:25.001 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:25.000421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:35.220 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:35.221154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:35.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:35.232981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:45.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:45.457024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:45.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:45.471600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:55.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:55.696502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:55.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:13:55.706148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:05.952 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:05.953423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:05.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:05.966666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:16.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:16.188604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:16.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:16.201714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:26.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:26.424254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:26.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:26.439382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:36.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:36.657041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:36.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:36.666842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:46.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:46.886294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:46.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:46.897978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:57.115 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:57.116099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:57.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:14:57.129189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:07.344 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:07.345299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:07.358 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:07.359739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:17.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:17.582091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:17.592 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:17.593926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:27.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:27.810818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:27.824 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:27.825974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:38.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:38.051037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:38.061 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:38.062696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:48.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:48.318476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:48.329 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:48.330728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:58.563 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:58.564920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:58.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:15:58.613537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:08.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:08.833699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:08.843 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:08.844700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:19.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:19.086262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:19.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:19.119326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:29.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:29.336741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:29.346 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:29.347579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:39.563 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:39.564229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:39.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:39.572815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:49.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:49.793277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:49.805 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:16:49.807095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:00.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:00.031961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:00.041 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:00.042984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:10.261 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:10.262931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:10.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:10.272584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:20.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:20.490766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:20.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:20.503282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:30.724 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:30.725286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:30.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:30.738663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:40.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:40.954424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:40.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:40.967470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:51.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:51.194541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:51.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:17:51.207020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:01.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:01.426356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:01.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:01.439701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:11.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:11.663490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:11.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:11.677631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:21.903 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:21.903812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:21.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:21.918325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:32.138 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:32.139078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:32.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:32.216754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:42.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:42.521341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:42.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:42.533205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:52.752 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:52.753170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:52.764 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:18:52.765259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:02.982 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:02.983304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:02.993 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:02.995253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:13.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:13.215665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:13.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:13.227752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:23.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:23.441600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:23.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:23.451442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:33.666 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:33.667329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:33.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:33.676007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:43.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:43.932022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:43.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:43.944145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:54.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:54.160087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:54.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:19:54.172427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:04.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:04.389197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:04.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:04.402353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:14.622 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:14.623217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:14.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:14.636280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:24.855 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:24.856041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:24.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:24.870432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:35.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:35.090128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:35.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:35.101100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:45.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:45.321924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:45.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:45.335028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:55.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:55.554477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:55.564 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:20:55.565868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:05.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:05.788029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:05.796 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:05.797320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:16.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:16.018839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:16.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:16.032318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:26.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:26.252388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:26.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:26.265109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:36.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:36.480050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:36.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:36.491698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:46.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:46.715403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:46.727 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:46.728309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:56.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:56.946613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:56.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:21:56.957675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:07.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:07.175549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:07.185 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:07.186714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:17.410 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:17.411909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:17.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:17.424297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:27.646 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:27.647536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:27.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:27.658113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:37.874 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:37.875939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:37.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:37.884723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:48.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:48.132304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:48.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:48.145087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:58.365 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:58.366405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:58.378 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:22:58.379850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:08.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:08.595416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:08.602 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:08.603628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:18.820 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:18.821512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:18.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:18.882549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:29.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:29.102444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:29.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:29.112757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:39.370 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:39.371786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:39.378 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:39.380519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:49.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:49.594774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:49.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:49.608655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:59.826 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:59.827432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:59.840 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:23:59.841130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:10.061 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:10.062066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:10.072 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:10.073333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:20.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:20.292340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:20.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:20.305199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:30.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:30.526660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:30.537 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:30.538683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:40.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:40.757825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:40.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:40.770084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:50.987 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:50.988998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:50.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:24:50.999535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:01.220 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:01.221738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:01.234 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:01.235420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:11.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:11.456145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:11.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:11.465239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:21.686 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:21.687787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:21.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:21.700185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:31.919 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:31.920802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:31.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:31.932948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:42.151 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:42.152882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:42.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:42.165320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:52.384 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:52.385922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:52.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:25:52.398161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:02.622 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:02.623668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:02.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:02.635678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:12.854 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:12.855390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:12.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:12.867658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:23.086 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:23.087728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:23.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:23.100489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:33.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:33.323237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:33.334 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:33.336434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:43.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:43.604421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:43.611 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:43.613083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:53.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:53.829157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:53.835 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:53.837298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:53.925 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:53.926960 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:26:53.917670709 +0000 UTC m=+12795.371948265 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:26:55.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:55.872153 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T02:26:56.496 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:56.497152 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:26:56.486+0000 7f15213f2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:26:56.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:26:56.486+0000 7f15213f2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:26:56.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:26:56.486+0000 7f15213f2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:26:56.497 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:26:56.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:26:56.497178 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:26:56.486+0000 7f15213f2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:26:56.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:26:56.486+0000 7f15213f2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:26:56.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:26:56.486+0000 7f15213f2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:26:56.498 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:27:04.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:04.054860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:04.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:04.067501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:14.286 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:14.287498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:14.299 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:14.300510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:24.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:24.566131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:24.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:24.606956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:34.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:34.843661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:34.851 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:34.853133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:45.073 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:45.074471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:45.086 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:45.087540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:55.304 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:55.305876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:55.315 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:27:55.316566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:05.530 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:05.531600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:05.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:05.547299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:15.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:15.769372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:15.781 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:15.783121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:26.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:26.017258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:26.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:26.030637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:36.250 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:36.251214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:36.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:36.259068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:46.490 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:46.491138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:46.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:46.501140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:46.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:46.501174 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:56.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:56.720526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:56.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:28:56.732710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:06.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:06.947198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:06.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:06.959818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:17.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:17.178215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:17.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:17.188720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:27.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:27.414475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:27.424 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:27.425899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:37.646 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:37.647336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:37.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:37.660547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:47.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:47.880120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:47.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:47.892544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:58.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:58.111949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:58.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:29:58.125561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:08.350 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:08.351802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:08.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:08.366212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:18.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:18.588584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:18.672 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:18.673917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:28.894 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:28.895757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:28.907 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:28.908598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:39.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:39.131757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:39.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:39.143295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:49.356 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:49.357333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:49.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:49.369213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:59.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:59.588465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:59.598 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:30:59.600222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:09.823 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:09.824895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:09.833 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:09.834615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:20.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:20.055285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:20.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:20.068021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:30.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:30.317188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:30.327 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:30.329369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:40.560 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:40.560473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:40.584 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:40.585558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:50.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:50.804534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:50.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:31:50.816667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:01.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:01.039137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:01.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:01.051341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:11.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:11.277833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:11.286 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:11.287470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:21.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:21.505301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:21.516 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:21.517347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:31.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:31.738084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:31.751 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:31.751192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:41.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:41.972886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:42.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:42.017437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:52.237 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:52.238974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:52.252 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:32:52.254231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:02.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:02.475795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:02.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:02.487277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:12.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:12.710929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:12.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:12.722255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:22.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:22.943117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:22.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:22.954468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:33.176 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:33.177035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:33.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:33.190978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:43.410 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:43.411131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:43.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:43.422649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:53.648 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:53.648989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:53.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:33:53.661017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:03.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:03.886202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:03.895 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:03.896737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:14.121 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:14.122215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:14.135 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:14.136597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:24.355 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:24.356258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:24.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:24.370122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:34.598 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:34.599519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:34.606 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:34.608207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:44.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:44.833801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:44.841 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:44.842546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:55.113 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:55.114212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:55.129 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:34:55.129914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:05.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:05.354751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:05.363 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:05.364407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:15.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:15.614879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:15.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:15.628743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:25.890 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:25.891106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:25.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:25.900492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:36.150 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:36.151420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:36.163 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:36.163844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:46.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:46.396213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:46.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:46.407110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:56.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:56.678265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:56.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:35:56.690650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:06.914 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:06.915692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:06.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:06.942046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:17.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:17.180733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:17.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:17.193272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:27.415 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:27.416721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:27.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:27.428409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:37.654 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:37.655552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:37.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:37.665709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:47.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:47.877333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:47.886 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:47.887896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:58.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:58.155986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:58.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:36:58.169859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:08.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:08.392028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:08.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:08.404165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:18.624 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:18.625497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:18.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:18.636612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:28.856 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:28.856847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:28.868 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:28.869357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:39.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:39.090168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:39.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:39.102513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:49.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:49.322979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:49.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:49.334103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:59.559 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:59.559905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:59.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:37:59.572653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:09.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:09.795196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:09.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:09.809153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:20.032 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:20.032910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:20.044 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:20.045447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:30.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:30.271089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:30.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:30.283060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:40.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:40.507256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:40.518 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:40.519600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:50.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:50.739345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:50.747 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:38:50.748731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:00.969 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:00.970370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:00.980 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:00.981415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:11.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:11.232383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:11.246 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:11.246605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:21.484 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:21.484772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:21.510 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:21.511260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:31.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:31.796177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:31.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:31.809037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:42.027 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:42.028089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:42.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:42.040427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:52.301 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:52.301955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:52.313 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:39:52.314629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:02.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:02.535403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:02.547 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:02.548648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:12.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:12.768431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:12.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:12.778776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:23.002 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:23.002517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:23.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:23.014754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:33.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:33.232988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:33.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:33.245019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:43.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:43.486042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:43.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:43.500091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:53.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:53.720664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:53.735 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:40:53.736025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:03.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:03.966398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:03.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:03.978137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:14.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:14.198015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:14.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:14.205770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:24.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:24.428262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:24.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:24.437312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:34.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:34.657103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:34.666 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:34.667795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:44.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:44.887997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:44.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:44.900675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:55.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:55.123684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:55.135 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:41:55.136069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:05.356 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:05.357059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:05.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:05.370337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:15.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:15.590752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:15.602 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:15.603037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:25.823 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:25.824008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:25.836 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:25.837125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:36.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:36.064566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:36.072 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:36.073523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:46.310 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:46.310411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:46.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:46.321978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:56.549 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:56.550026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:56.560 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:42:56.559539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:06.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:06.781437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:06.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:06.795144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:17.036 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:17.036642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:17.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:17.051232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:27.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:27.270601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:27.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:27.284404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:36.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:36.504567 I | op-mon: parsing mon endpoints: a=10.107.60.203:6789 2022-03-09T02:43:37.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:37.143895 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:43:37.131+0000 7f1f9851a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:43:37.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:43:37.131+0000 7f1f9851a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:43:37.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:43:37.131+0000 7f1f9851a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:43:37.144 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:43:37.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:37.143931 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:43:37.131+0000 7f1f9851a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:43:37.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:43:37.131+0000 7f1f9851a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:43:37.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09T02:43:37.131+0000 7f1f9851a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:43:37.145 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:43:37.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:37.500738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:37.513 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:37.513333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:47.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:47.733053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:47.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:47.783708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:58.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:58.007619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:58.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:43:58.021097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:08.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:08.241743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:08.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:08.252485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:18.488 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:18.489115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:18.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:18.504337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:28.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:28.722107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:28.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:28.734955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:38.965 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:38.965517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:38.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:38.974053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:49.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:49.190359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:49.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:49.201324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:59.417 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:59.418243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:59.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:44:59.429412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:09.643 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:09.644254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:09.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:09.655924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:09.855 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:09.856560 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:45:09.846292882 +0000 UTC m=+13891.300570435 LastTransitionTime:2022-03-08 22:55:33 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:45:19.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:19.872807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:19.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:19.886519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:30.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:30.107272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:30.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:30.121188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:40.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:40.342668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:40.350 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:40.351164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:50.572 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:50.572876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:50.586 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:45:50.587669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:00.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:00.808899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:00.820 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:00.821363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:11.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:11.039046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:11.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:11.051791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:21.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:21.271887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:21.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:21.283925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:31.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:31.508162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:31.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:31.521260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:41.741 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:41.741733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:41.750 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:41.751585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:51.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:51.974652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:52.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:46:52.022599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:02.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:02.241799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:02.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:02.255684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:12.475 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:12.475973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:12.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:12.547372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:22.776 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:22.777398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:22.785 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:22.786316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:33.001 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:33.001435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:33.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:33.011891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:43.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:43.231414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:43.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:43.245234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:53.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:53.480215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:53.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:47:53.494096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:03.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:03.716871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:03.730 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:03.730339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:13.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:13.949330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:13.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:13.962750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:24.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:24.215105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:24.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:24.228475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:34.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:34.450235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:34.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:34.462927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:44.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:44.681625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:44.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:44.695425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:54.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:54.917746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:54.970 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:48:54.971049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:05.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:05.190137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:05.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:05.203783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:15.426 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:15.426493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:15.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:15.440988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:25.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:25.661584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:25.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:25.674869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:35.902 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:35.899734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:35.908 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:35.908693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:46.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:46.137395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:46.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:46.157998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:56.383 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:56.383294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:56.394 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:49:56.394767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:06.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:06.644402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:06.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:06.659360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:16.877 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:16.877614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:16.889 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:16.889687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:27.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:27.112520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:27.135 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:27.136207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:37.362 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:37.362351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:37.373 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:37.374355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:47.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:47.596339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:47.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:47.609283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:57.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:57.832625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:57.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:50:57.844723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:08.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:08.077829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:08.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:08.086299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:18.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:18.317918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:18.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:18.329000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:28.545 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:28.545728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:28.556 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:28.557443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:38.778 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:38.779137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:38.793 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:38.794076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:49.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:49.012191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:49.083 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:49.083523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:59.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:59.312104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:59.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:51:59.322999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:09.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:09.541087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:09.554 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:09.554901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:19.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:19.769019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:19.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:19.781191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:30.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:30.000803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:30.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:30.034600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:40.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:40.256604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:40.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:40.268026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:50.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:50.488139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:50.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:52:50.500322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:00.736 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:00.736721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:00.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:00.750340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:10.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:10.973062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:10.988 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:10.987698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:21.202 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:21.202539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:21.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:21.214823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:31.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:31.432420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:31.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:31.442351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:41.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:41.661815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:41.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-09 02:53:41.670353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T05:14:38.278 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-09T05:14:38.305 DEBUG:teuthology.task.console_log:Killing console logger for smithi186 2022-03-09T05:14:38.307 DEBUG:teuthology.exit:Finished running handlers