2022-03-07T17:26:02.500 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-07T17:26:02.502 DEBUG:teuthology.run:Teuthology command: teuthology --name lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/1.7.2} --archive /home/teuthworker/archive/lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi/6725083 --verbose --owner scheduled_lflores@teuthology -- /home/teuthworker/archive/lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi/6725083/orig.config.yaml 2022-03-07T17:26:02.545 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-07T17:26:02.608 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi/6725083 branch: balancer-perf description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/1.7.2} email: null first_in_suite: false job_id: '6725083' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: balancer-perf ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: balancer-perf sha1: 27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 owner: scheduled_lflores@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 sleep_before_teardown: 0 suite: rados suite_branch: balancer-perf suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 targets: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2cu5oPA622UOvbafmAL6OKWytCltPfuR5V0aHeVs/eNXpN/sYTnDa3QIedNGtp5WFAkFdLRpe0Rghw49aNIIogb/QgZ28MX51MJOojaEp+zn7PEZR2q9LY+qD7/xg3IA2ll/5w+PAj2HJbaMEh1efzaZ5ibkX/yZzMAiHHmGKAyDS5T9naYEo99yjUDdNueH7ABN10NqcHXjvqbuhY3ZouZqhniuyP7Ni/jRI1Chyev2WDTrZ28wUBWxWMK4UTYm2m9IQLy3FwLTcupEazwrstWc5E5mSC0WwNe2OqbOgtcejUw2pnDkVjhwkbVhVXFDficNCsT0R6+heFBP2t1RCbGbhOJx1y6J/XWp7tXRvxYNJdaO41WucCLfxmDHdFNQJBfNhQHjYtB0Sg7gSBwMPngl/oNayFUWQUWxnh21o/AdXl/i18dQzfaNKfiVzsBeOF4UoIZkTFKnP9jKMHfThwKR6CYz6uYTO/k6wgGSruorbxpeV6fhQWrUXN5kEznc= smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/UObH6gSKZMsWL7A2YG2HwFWchBw8vmB9bqBu8H2Po/OiTlp0RjuwJikMxc+viiVLvTJD+20WJ0Cq8pRIpelLaYIn83lT9K/gLG55COWiAUd6vgQmnrpYmkkoTbAM2aSJzBiDaHahU3+zDKebcc9SlHtfQr5+bnS+TUZRwP4lvZ8VyGeR9oSS3qklkhazsfM29EPPB74F0CD12kPBOedMul1tBfkDoT25r+/FaebpDwweHlTQsnL6Gt4bDP/nh+IqyKBVCbcQBQzjWTbTf1/2GSuSRk3IcwP0NOzCl7oUAmD2khYxMSyLaBVbchrGb1yRssbmh5PCcRKnaH822822JOpbvctqlCkjPj1N/z/xx22R0WLKi3kHRN2SjM898wuVL7cEsFVa5qkrz57Y9/XMtC7I8VJemrF5kymqKaFmaF5ydzKpcjf123MTwaaZ0SZQiqk6p5BST9PkDyTl19N1+7BV70s/TDWRgyw1MJFO6qVhhiFD1E6BEHcCo1HaBAc= smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvoL62sXPEloyYQv9HR9jl7Cjw5mhNyLYINq7SR/WU60dx0YUUGgVXS7fjUzLIQYg1LTRU7iP27JOHidsFyVQcauOJ6bNiLyu+hmc82hVFk62nczrXW49ELPXVSUPcAgtGtgPjRSBumaPM+87/5K5/eJjUcuIE31t0HXmYHFny3uW+yc+3QqfhUBESAcWtNmWQG2flh8+JbL5pqRKutoNP09wcz4hYjyXFensnkrJ4Zw894s8PmZM7VebKmYWqyzNq3fjR7QswAhtJZHMfOQx2wD7zVquIdZmJJOXDE/5MzrFO75KBkN/h+a7e0/TBPt3gaDJvP57iAtx6Tkun19x+1Nh4rXXzwQCyzCDgcJ2wX1OUslOzOefrZl/4PSv+KcqJGVWypg9tleBBE2kgaK9Pa9Sr2zdEgtYmsR4a8qH8h2G+x1ymBMQZ+Y6Wba7JMjxhhRcEueg5B8QW6BG7/Bxyexl5PgJpc395EDgfRl6KmJAcD2Ev6bMAIoWiLlyOiAc= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-07_17:16:09 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-07T17:26:02.608 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa; will attempt to use it 2022-03-07T17:26:02.609 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks 2022-03-07T17:26:02.610 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-07T17:26:02.612 INFO:teuthology.task.internal:Checking packages... 2022-03-07T17:26:02.633 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2' 2022-03-07T17:26:02.633 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-07T17:26:02.634 INFO:teuthology.packaging:ref: None 2022-03-07T17:26:02.634 INFO:teuthology.packaging:tag: None 2022-03-07T17:26:02.634 INFO:teuthology.packaging:branch: balancer-perf 2022-03-07T17:26:02.635 INFO:teuthology.packaging:sha1: 27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 2022-03-07T17:26:02.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=balancer-perf 2022-03-07T17:26:02.792 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10890-g27130916-1focal 2022-03-07T17:26:02.792 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-07T17:26:02.818 INFO:teuthology.task.internal:no buildpackages task found 2022-03-07T17:26:02.818 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-07T17:26:02.856 INFO:teuthology.task.internal:Saving configuration 2022-03-07T17:26:02.870 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-07T17:26:02.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-07T17:26:02.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi/6725083', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-07 17:19:02.498814', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKE0mwbhwhOy3L671GZ66ERBrjJf2X7NH6IvCUgfH1fs'} 2022-03-07T17:26:02.935 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi/6725083', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-07 17:19:02.497745', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPDvyAaK0RLtA7LMkQ3xtVgkrOhXp7snq6Imef8zXIzT'} 2022-03-07T17:26:02.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi085.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-03-07_17:16:09-rados-balancer-perf-distro-default-smithi/6725083', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-07 17:19:02.496392', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB6SO6e/+KhrcR9TipwKhMS2Taq4CnJjHcbQ594QLHwk'} 2022-03-07T17:26:02.955 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-07T17:26:02.969 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-07T17:26:02.969 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-07T17:26:02.970 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-07T17:26:02.970 INFO:teuthology.run_tasks:Running task console_log... 2022-03-07T17:26:03.097 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff4f5e36950>, signals=[15]) 2022-03-07T17:26:03.099 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-07T17:26:03.113 INFO:teuthology.task.internal:Opening connections... 2022-03-07T17:26:03.113 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2022-03-07T17:26:03.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:26:03.167 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2022-03-07T17:26:03.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:26:03.222 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi085.front.sepia.ceph.com 2022-03-07T17:26:03.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:26:03.270 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-07T17:26:03.284 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2022-03-07T17:26:03.291 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2022-03-07T17:26:03.292 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2022-03-07T17:26:03.340 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Ubuntu" 2022-03-07T17:26:03.341 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-07T17:26:03.341 INFO:teuthology.orchestra.run.smithi057.stdout:ID=ubuntu 2022-03-07T17:26:03.341 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE=debian 2022-03-07T17:26:03.342 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-07T17:26:03.342 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="20.04" 2022-03-07T17:26:03.342 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-07T17:26:03.342 INFO:teuthology.orchestra.run.smithi057.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-07T17:26:03.343 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-07T17:26:03.343 INFO:teuthology.orchestra.run.smithi057.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-07T17:26:03.343 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_CODENAME=focal 2022-03-07T17:26:03.343 INFO:teuthology.orchestra.run.smithi057.stdout:UBUNTU_CODENAME=focal 2022-03-07T17:26:03.344 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2022-03-07T17:26:03.362 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2022-03-07T17:26:03.369 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2022-03-07T17:26:03.370 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2022-03-07T17:26:03.417 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="Ubuntu" 2022-03-07T17:26:03.417 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-07T17:26:03.418 INFO:teuthology.orchestra.run.smithi066.stdout:ID=ubuntu 2022-03-07T17:26:03.418 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE=debian 2022-03-07T17:26:03.418 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-07T17:26:03.418 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="20.04" 2022-03-07T17:26:03.419 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-07T17:26:03.419 INFO:teuthology.orchestra.run.smithi066.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-07T17:26:03.419 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-07T17:26:03.419 INFO:teuthology.orchestra.run.smithi066.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-07T17:26:03.420 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_CODENAME=focal 2022-03-07T17:26:03.420 INFO:teuthology.orchestra.run.smithi066.stdout:UBUNTU_CODENAME=focal 2022-03-07T17:26:03.420 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2022-03-07T17:26:03.438 DEBUG:teuthology.orchestra.run.smithi085:> uname -m 2022-03-07T17:26:03.445 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2022-03-07T17:26:03.446 DEBUG:teuthology.orchestra.run.smithi085:> cat /etc/os-release 2022-03-07T17:26:03.493 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Ubuntu" 2022-03-07T17:26:03.494 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-07T17:26:03.494 INFO:teuthology.orchestra.run.smithi085.stdout:ID=ubuntu 2022-03-07T17:26:03.494 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE=debian 2022-03-07T17:26:03.494 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-07T17:26:03.495 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="20.04" 2022-03-07T17:26:03.495 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-07T17:26:03.495 INFO:teuthology.orchestra.run.smithi085.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-07T17:26:03.495 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-07T17:26:03.496 INFO:teuthology.orchestra.run.smithi085.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-07T17:26:03.496 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_CODENAME=focal 2022-03-07T17:26:03.496 INFO:teuthology.orchestra.run.smithi085.stdout:UBUNTU_CODENAME=focal 2022-03-07T17:26:03.497 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2022-03-07T17:26:03.514 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-07T17:26:03.531 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-07T17:26:03.565 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-07T17:26:03.565 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2022-03-07T17:26:03.568 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2022-03-07T17:26:03.570 DEBUG:teuthology.orchestra.run.smithi085:> test '!' -e /home/ubuntu/cephtest 2022-03-07T17:26:03.577 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-07T17:26:03.596 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-07T17:26:03.597 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2022-03-07T17:26:03.599 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2022-03-07T17:26:03.617 DEBUG:teuthology.orchestra.run.smithi085:> test -z $(ls -A /var/lib/ceph) 2022-03-07T17:26:03.929 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-07T17:26:04.004 INFO:teuthology.run_tasks:Running task kernel... 2022-03-07T17:26:04.024 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-07T17:26:04.025 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-07T17:26:04.025 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-07T17:26:04.026 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-07T17:26:04.026 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2022-03-07T17:26:04.027 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2022-03-07T17:26:04.028 DEBUG:teuthology.orchestra.run.smithi085:> uname -r 2022-03-07T17:26:04.035 INFO:teuthology.orchestra.run.smithi066.stdout:5.4.0-97-generic 2022-03-07T17:26:04.036 INFO:teuthology.orchestra.run.smithi057.stdout:5.4.0-97-generic 2022-03-07T17:26:04.036 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.4.0-97-generic 2022-03-07T17:26:04.036 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get clean 2022-03-07T17:26:04.037 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.4.0-97-generic 2022-03-07T17:26:04.037 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get clean 2022-03-07T17:26:04.039 INFO:teuthology.orchestra.run.smithi085.stdout:5.4.0-97-generic 2022-03-07T17:26:04.039 INFO:teuthology.task.kernel:Running kernel on smithi085: 5.4.0-97-generic 2022-03-07T17:26:04.039 DEBUG:teuthology.orchestra.run.smithi085:> sudo apt-get clean 2022-03-07T17:26:04.117 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get update 2022-03-07T17:26:04.146 DEBUG:teuthology.orchestra.run.smithi085:> sudo apt-get update 2022-03-07T17:26:04.154 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2022-03-07T17:26:04.354 INFO:teuthology.orchestra.run.smithi085.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:26:04.366 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:26:04.448 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:26:04.449 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:26:04.492 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:26:04.498 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:26:04.519 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:26:04.558 INFO:teuthology.orchestra.run.smithi085.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:26:04.562 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:26:04.881 INFO:teuthology.orchestra.run.smithi085.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-07T17:26:04.911 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-07T17:26:04.912 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:26:04.958 INFO:teuthology.orchestra.run.smithi085.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:26:04.964 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:26:04.989 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-07T17:26:05.026 INFO:teuthology.orchestra.run.smithi085.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-07T17:26:05.113 INFO:teuthology.orchestra.run.smithi085.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-07T17:26:05.126 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-07T17:26:05.132 INFO:teuthology.orchestra.run.smithi085.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-07T17:26:05.135 INFO:teuthology.orchestra.run.smithi085.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-07T17:26:05.136 INFO:teuthology.orchestra.run.smithi085.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-07T17:26:05.136 INFO:teuthology.orchestra.run.smithi085.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-07T17:26:05.137 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-07T17:26:05.138 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-07T17:26:05.148 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-07T17:26:05.155 INFO:teuthology.orchestra.run.smithi085.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-07T17:26:05.157 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-07T17:26:05.158 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-07T17:26:05.158 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-07T17:26:05.166 INFO:teuthology.orchestra.run.smithi085.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-07T17:26:05.166 INFO:teuthology.orchestra.run.smithi085.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-07T17:26:05.175 INFO:teuthology.orchestra.run.smithi085.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-07T17:26:05.182 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-07T17:26:05.185 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-07T17:26:05.186 INFO:teuthology.orchestra.run.smithi057.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-07T17:26:05.190 INFO:teuthology.orchestra.run.smithi085.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-07T17:26:05.191 INFO:teuthology.orchestra.run.smithi085.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-07T17:26:05.203 INFO:teuthology.orchestra.run.smithi057.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-07T17:26:05.210 INFO:teuthology.orchestra.run.smithi057.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-07T17:26:05.211 INFO:teuthology.orchestra.run.smithi085.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-07T17:26:05.212 INFO:teuthology.orchestra.run.smithi057.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-07T17:26:05.212 INFO:teuthology.orchestra.run.smithi085.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-07T17:26:05.212 INFO:teuthology.orchestra.run.smithi085.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-07T17:26:05.213 INFO:teuthology.orchestra.run.smithi085.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-07T17:26:05.223 INFO:teuthology.orchestra.run.smithi057.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-07T17:26:05.223 INFO:teuthology.orchestra.run.smithi057.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-07T17:26:05.224 INFO:teuthology.orchestra.run.smithi057.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-07T17:26:05.224 INFO:teuthology.orchestra.run.smithi057.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-07T17:26:05.409 INFO:teuthology.orchestra.run.smithi057.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-07T17:26:05.479 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-07T17:26:05.498 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-07T17:26:05.507 INFO:teuthology.orchestra.run.smithi085.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-07T17:26:05.511 INFO:teuthology.orchestra.run.smithi066.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-07T17:26:05.512 INFO:teuthology.orchestra.run.smithi066.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-07T17:26:05.513 INFO:teuthology.orchestra.run.smithi066.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-07T17:26:05.522 INFO:teuthology.orchestra.run.smithi066.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-07T17:26:05.523 INFO:teuthology.orchestra.run.smithi085.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-07T17:26:05.529 INFO:teuthology.orchestra.run.smithi085.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-07T17:26:05.530 INFO:teuthology.orchestra.run.smithi085.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-07T17:26:05.530 INFO:teuthology.orchestra.run.smithi085.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-07T17:26:05.543 INFO:teuthology.orchestra.run.smithi066.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-07T17:26:05.551 INFO:teuthology.orchestra.run.smithi066.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-07T17:26:05.552 INFO:teuthology.orchestra.run.smithi066.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-07T17:26:05.579 INFO:teuthology.orchestra.run.smithi057.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-07T17:26:05.586 INFO:teuthology.orchestra.run.smithi066.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-07T17:26:05.589 INFO:teuthology.orchestra.run.smithi066.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-07T17:26:05.590 INFO:teuthology.orchestra.run.smithi066.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-07T17:26:05.590 INFO:teuthology.orchestra.run.smithi057.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-07T17:26:05.591 INFO:teuthology.orchestra.run.smithi057.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-07T17:26:05.603 INFO:teuthology.orchestra.run.smithi085.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-07T17:26:05.606 INFO:teuthology.orchestra.run.smithi085.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-07T17:26:05.606 INFO:teuthology.orchestra.run.smithi085.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-07T17:26:05.619 INFO:teuthology.orchestra.run.smithi066.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-07T17:26:05.620 INFO:teuthology.orchestra.run.smithi066.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-07T17:26:05.620 INFO:teuthology.orchestra.run.smithi085.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-07T17:26:05.623 INFO:teuthology.orchestra.run.smithi066.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-07T17:26:05.624 INFO:teuthology.orchestra.run.smithi066.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-07T17:26:05.628 INFO:teuthology.orchestra.run.smithi066.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-07T17:26:05.641 INFO:teuthology.orchestra.run.smithi057.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-07T17:26:05.642 INFO:teuthology.orchestra.run.smithi066.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-07T17:26:05.643 INFO:teuthology.orchestra.run.smithi057.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-07T17:26:05.653 INFO:teuthology.orchestra.run.smithi057.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-07T17:26:05.654 INFO:teuthology.orchestra.run.smithi057.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-07T17:26:05.658 INFO:teuthology.orchestra.run.smithi085.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-07T17:26:05.666 INFO:teuthology.orchestra.run.smithi085.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-07T17:26:05.667 INFO:teuthology.orchestra.run.smithi066.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-07T17:26:05.669 INFO:teuthology.orchestra.run.smithi066.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-07T17:26:05.671 INFO:teuthology.orchestra.run.smithi066.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-07T17:26:05.676 INFO:teuthology.orchestra.run.smithi066.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-07T17:26:05.694 INFO:teuthology.orchestra.run.smithi057.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-07T17:26:05.700 INFO:teuthology.orchestra.run.smithi066.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-07T17:26:05.700 INFO:teuthology.orchestra.run.smithi066.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-07T17:26:05.701 INFO:teuthology.orchestra.run.smithi066.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-07T17:26:05.701 INFO:teuthology.orchestra.run.smithi066.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-07T17:26:05.708 INFO:teuthology.orchestra.run.smithi057.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-07T17:26:05.711 INFO:teuthology.orchestra.run.smithi057.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-07T17:26:05.712 INFO:teuthology.orchestra.run.smithi085.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-07T17:26:05.712 INFO:teuthology.orchestra.run.smithi085.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,456 B] 2022-03-07T17:26:05.713 INFO:teuthology.orchestra.run.smithi085.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-07T17:26:05.713 INFO:teuthology.orchestra.run.smithi085.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-07T17:26:05.714 INFO:teuthology.orchestra.run.smithi085.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-07T17:26:05.715 INFO:teuthology.orchestra.run.smithi085.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-07T17:26:05.722 INFO:teuthology.orchestra.run.smithi066.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-07T17:26:05.723 INFO:teuthology.orchestra.run.smithi066.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,456 B] 2022-03-07T17:26:05.724 INFO:teuthology.orchestra.run.smithi066.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-07T17:26:05.724 INFO:teuthology.orchestra.run.smithi066.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-07T17:26:05.725 INFO:teuthology.orchestra.run.smithi066.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-07T17:26:05.728 INFO:teuthology.orchestra.run.smithi066.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-07T17:26:05.729 INFO:teuthology.orchestra.run.smithi066.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-07T17:26:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,456 B] 2022-03-07T17:26:05.739 INFO:teuthology.orchestra.run.smithi057.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-07T17:26:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-07T17:26:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-07T17:26:05.741 INFO:teuthology.orchestra.run.smithi057.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-07T17:26:05.787 INFO:teuthology.orchestra.run.smithi057.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-07T17:26:05.798 INFO:teuthology.orchestra.run.smithi085.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-07T17:26:05.799 INFO:teuthology.orchestra.run.smithi085.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-07T17:26:05.799 INFO:teuthology.orchestra.run.smithi085.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-07T17:26:05.799 INFO:teuthology.orchestra.run.smithi085.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-07T17:26:05.800 INFO:teuthology.orchestra.run.smithi085.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-07T17:26:05.810 INFO:teuthology.orchestra.run.smithi066.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-07T17:26:05.811 INFO:teuthology.orchestra.run.smithi066.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-07T17:26:05.812 INFO:teuthology.orchestra.run.smithi066.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-07T17:26:05.812 INFO:teuthology.orchestra.run.smithi066.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-07T17:26:05.875 INFO:teuthology.orchestra.run.smithi057.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-07T17:26:05.876 INFO:teuthology.orchestra.run.smithi057.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-07T17:26:05.878 INFO:teuthology.orchestra.run.smithi057.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-07T17:26:05.881 INFO:teuthology.orchestra.run.smithi057.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-07T17:26:05.882 INFO:teuthology.orchestra.run.smithi057.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-07T17:26:08.215 INFO:teuthology.orchestra.run.smithi085.stdout:Fetched 10.0 MB in 2s (5,084 kB/s) 2022-03-07T17:26:08.480 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 10.0 MB in 2s (4,932 kB/s) 2022-03-07T17:26:08.723 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 10.0 MB in 2s (4,296 kB/s) 2022-03-07T17:26:09.417 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:26:09.439 DEBUG:teuthology.orchestra.run.smithi085:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T17:26:09.511 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:26:09.684 INFO:teuthology.orchestra.run.smithi085.stdout:Building dependency tree... 2022-03-07T17:26:09.684 INFO:teuthology.orchestra.run.smithi085.stdout:Reading state information... 2022-03-07T17:26:09.752 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:26:09.776 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T17:26:09.829 INFO:teuthology.orchestra.run.smithi085.stdout:The following additional packages will be installed: 2022-03-07T17:26:09.830 INFO:teuthology.orchestra.run.smithi085.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T17:26:09.830 INFO:teuthology.orchestra.run.smithi085.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-07T17:26:09.830 INFO:teuthology.orchestra.run.smithi085.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-07T17:26:09.831 INFO:teuthology.orchestra.run.smithi085.stdout:Suggested packages: 2022-03-07T17:26:09.831 INFO:teuthology.orchestra.run.smithi085.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-07T17:26:09.831 INFO:teuthology.orchestra.run.smithi085.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-07T17:26:09.848 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:26:09.876 INFO:teuthology.orchestra.run.smithi085.stdout:The following NEW packages will be installed: 2022-03-07T17:26:09.876 INFO:teuthology.orchestra.run.smithi085.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T17:26:09.876 INFO:teuthology.orchestra.run.smithi085.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-07T17:26:09.877 INFO:teuthology.orchestra.run.smithi085.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-07T17:26:09.877 INFO:teuthology.orchestra.run.smithi085.stdout: thermald upower usbmuxd 2022-03-07T17:26:09.927 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:26:09.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T17:26:10.022 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:26:10.029 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-07T17:26:10.030 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-07T17:26:10.056 INFO:teuthology.orchestra.run.smithi085.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T17:26:10.057 INFO:teuthology.orchestra.run.smithi085.stdout:Need to get 80.1 MB of archives. 2022-03-07T17:26:10.057 INFO:teuthology.orchestra.run.smithi085.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-07T17:26:10.057 INFO:teuthology.orchestra.run.smithi085.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-07T17:26:10.179 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2022-03-07T17:26:10.179 INFO:teuthology.orchestra.run.smithi057.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T17:26:10.179 INFO:teuthology.orchestra.run.smithi057.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-07T17:26:10.179 INFO:teuthology.orchestra.run.smithi057.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-07T17:26:10.180 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2022-03-07T17:26:10.180 INFO:teuthology.orchestra.run.smithi057.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-07T17:26:10.180 INFO:teuthology.orchestra.run.smithi057.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-07T17:26:10.199 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-07T17:26:10.199 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-07T17:26:10.231 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2022-03-07T17:26:10.232 INFO:teuthology.orchestra.run.smithi057.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T17:26:10.232 INFO:teuthology.orchestra.run.smithi057.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-07T17:26:10.233 INFO:teuthology.orchestra.run.smithi057.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-07T17:26:10.233 INFO:teuthology.orchestra.run.smithi057.stdout: thermald upower usbmuxd 2022-03-07T17:26:10.321 INFO:teuthology.orchestra.run.smithi085.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-07T17:26:10.347 INFO:teuthology.orchestra.run.smithi066.stdout:The following additional packages will be installed: 2022-03-07T17:26:10.347 INFO:teuthology.orchestra.run.smithi066.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T17:26:10.347 INFO:teuthology.orchestra.run.smithi066.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-07T17:26:10.347 INFO:teuthology.orchestra.run.smithi066.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-07T17:26:10.348 INFO:teuthology.orchestra.run.smithi066.stdout:Suggested packages: 2022-03-07T17:26:10.348 INFO:teuthology.orchestra.run.smithi066.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-07T17:26:10.348 INFO:teuthology.orchestra.run.smithi066.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-07T17:26:10.360 INFO:teuthology.orchestra.run.smithi085.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-07T17:26:10.360 INFO:teuthology.orchestra.run.smithi085.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-07T17:26:10.400 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2022-03-07T17:26:10.401 INFO:teuthology.orchestra.run.smithi066.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T17:26:10.401 INFO:teuthology.orchestra.run.smithi066.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-07T17:26:10.401 INFO:teuthology.orchestra.run.smithi066.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-07T17:26:10.402 INFO:teuthology.orchestra.run.smithi066.stdout: thermald upower usbmuxd 2022-03-07T17:26:10.416 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T17:26:10.416 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 80.1 MB of archives. 2022-03-07T17:26:10.416 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-07T17:26:10.416 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-07T17:26:10.425 INFO:teuthology.orchestra.run.smithi085.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-07T17:26:10.461 INFO:teuthology.orchestra.run.smithi085.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-07T17:26:10.586 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T17:26:10.587 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 80.1 MB of archives. 2022-03-07T17:26:10.587 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-07T17:26:10.587 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-07T17:26:10.722 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-07T17:26:10.761 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-07T17:26:10.787 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-07T17:26:10.867 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-07T17:26:10.886 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-07T17:26:10.894 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-07T17:26:10.895 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-07T17:26:10.901 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-07T17:26:10.979 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-07T17:26:10.980 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-07T17:26:11.331 INFO:teuthology.orchestra.run.smithi085.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-07T17:26:11.660 INFO:teuthology.orchestra.run.smithi085.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-07T17:26:12.046 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-07T17:26:12.191 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-07T17:26:12.364 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-07T17:26:12.550 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-07T17:26:13.755 INFO:teuthology.orchestra.run.smithi085.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-07T17:26:13.755 INFO:teuthology.orchestra.run.smithi085.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-07T17:26:13.756 INFO:teuthology.orchestra.run.smithi085.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-07T17:26:13.761 INFO:teuthology.orchestra.run.smithi085.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-07T17:26:13.764 INFO:teuthology.orchestra.run.smithi085.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-07T17:26:14.169 INFO:teuthology.orchestra.run.smithi085.stdout:Fetched 80.1 MB in 4s (20.6 MB/s) 2022-03-07T17:26:14.380 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-07T17:26:14.552 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-07T17:26:14.552 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-07T17:26:14.553 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-07T17:26:14.558 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-07T17:26:14.612 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-07T17:26:14.801 INFO:teuthology.orchestra.run.smithi066.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-07T17:26:14.801 INFO:teuthology.orchestra.run.smithi066.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-07T17:26:14.802 INFO:teuthology.orchestra.run.smithi066.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-07T17:26:14.808 INFO:teuthology.orchestra.run.smithi066.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-07T17:26:14.810 INFO:teuthology.orchestra.run.smithi066.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-07T17:26:15.152 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 80.1 MB in 4s (18.3 MB/s) 2022-03-07T17:26:15.251 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 80.1 MB in 4s (18.2 MB/s) 2022-03-07T17:26:15.384 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-07T17:26:15.411 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-07T17:26:15.872 INFO:teuthology.orchestra.run.smithi085.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-07T17:26:15.876 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-07T17:26:15.993 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T17:26:16.321 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-07T17:26:16.337 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-07T17:26:16.379 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T17:26:16.732 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-07T17:26:16.744 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-07T17:26:16.782 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T17:26:16.892 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-07T17:26:16.897 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-07T17:26:16.901 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-07T17:26:16.907 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-07T17:26:17.041 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T17:26:17.109 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-07T17:26:17.114 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T17:26:17.124 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-07T17:26:17.168 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T17:26:17.444 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-07T17:26:17.460 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-07T17:26:17.479 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-07T17:26:17.492 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-07T17:26:17.502 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T17:26:17.529 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T17:26:17.551 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-07T17:26:17.565 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-07T17:26:17.609 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T17:26:17.822 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-07T17:26:17.831 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-07T17:26:17.835 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-07T17:26:17.846 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:17.864 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T17:26:17.890 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:17.996 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-07T17:26:18.012 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-07T17:26:18.054 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T17:26:18.242 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-07T17:26:18.255 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-07T17:26:18.300 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T17:26:18.391 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-07T17:26:18.405 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-07T17:26:18.449 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T17:26:18.678 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-07T17:26:18.690 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-07T17:26:18.744 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T17:26:18.852 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-07T17:26:18.866 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-07T17:26:18.910 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T17:26:19.047 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-07T17:26:19.062 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:19.097 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:19.255 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-07T17:26:19.271 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:19.314 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:20.039 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-07T17:26:20.053 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:20.159 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:20.667 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-07T17:26:20.680 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:20.725 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:21.456 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-07T17:26:21.470 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:21.557 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:21.799 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-07T17:26:21.812 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:21.931 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:22.043 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-07T17:26:22.056 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:22.101 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:22.469 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-07T17:26:22.483 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T17:26:22.552 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:26.589 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-07T17:26:26.604 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-07T17:26:26.645 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T17:26:26.998 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-07T17:26:27.014 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-07T17:26:27.056 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T17:26:27.401 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package thermald. 2022-03-07T17:26:27.417 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-07T17:26:27.451 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T17:26:27.934 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-07T17:26:27.952 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-07T17:26:27.975 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package upower. 2022-03-07T17:26:27.988 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-07T17:26:27.992 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T17:26:28.043 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-07T17:26:28.348 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-07T17:26:28.366 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-07T17:26:28.423 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T17:26:28.459 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package usbmuxd. 2022-03-07T17:26:28.473 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-07T17:26:28.508 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T17:26:28.603 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-07T17:26:28.620 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-07T17:26:28.653 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T17:26:28.790 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T17:26:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package thermald. 2022-03-07T17:26:28.824 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-07T17:26:28.860 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T17:26:28.913 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T17:26:28.980 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-07T17:26:28.993 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-07T17:26:29.014 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T17:26:29.038 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T17:26:29.148 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T17:26:29.274 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T17:26:29.400 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T17:26:29.434 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package upower. 2022-03-07T17:26:29.449 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-07T17:26:29.449 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package thermald. 2022-03-07T17:26:29.464 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-07T17:26:29.502 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-07T17:26:29.508 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T17:26:29.526 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-07T17:26:29.951 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package usbmuxd. 2022-03-07T17:26:29.965 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-07T17:26:29.996 INFO:teuthology.orchestra.run.smithi085.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-07T17:26:30.034 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T17:26:30.044 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T17:26:30.090 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package upower. 2022-03-07T17:26:30.106 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-07T17:26:30.158 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-07T17:26:30.210 INFO:teuthology.orchestra.run.smithi085.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-07T17:26:30.220 INFO:teuthology.orchestra.run.smithi085.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-07T17:26:30.221 INFO:teuthology.orchestra.run.smithi085.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-07T17:26:30.314 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T17:26:30.455 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T17:26:30.581 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T17:26:30.598 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package usbmuxd. 2022-03-07T17:26:30.612 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-07T17:26:30.669 INFO:teuthology.orchestra.run.smithi085.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-07T17:26:30.682 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T17:26:30.707 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T17:26:30.808 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T17:26:30.900 INFO:teuthology.orchestra.run.smithi085.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-07T17:26:30.934 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T17:26:30.965 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T17:26:30.997 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T17:26:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-07T17:26:31.144 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T17:26:31.270 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T17:26:31.381 INFO:teuthology.orchestra.run.smithi085.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T17:26:31.382 INFO:teuthology.orchestra.run.smithi085.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T17:26:31.429 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T17:26:31.550 INFO:teuthology.orchestra.run.smithi066.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-07T17:26:31.572 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T17:26:31.628 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T17:26:31.698 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T17:26:31.807 INFO:teuthology.orchestra.run.smithi066.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-07T17:26:31.816 INFO:teuthology.orchestra.run.smithi066.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-07T17:26:31.816 INFO:teuthology.orchestra.run.smithi066.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-07T17:26:31.849 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-07T17:26:31.921 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:32.296 INFO:teuthology.orchestra.run.smithi057.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-07T17:26:32.327 INFO:teuthology.orchestra.run.smithi066.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-07T17:26:32.342 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T17:26:32.527 INFO:teuthology.orchestra.run.smithi057.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-07T17:26:32.533 INFO:teuthology.orchestra.run.smithi057.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-07T17:26:32.533 INFO:teuthology.orchestra.run.smithi057.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-07T17:26:32.571 INFO:teuthology.orchestra.run.smithi066.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-07T17:26:32.616 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T17:26:32.928 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T17:26:32.929 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T17:26:33.017 INFO:teuthology.orchestra.run.smithi057.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-07T17:26:33.185 INFO:teuthology.orchestra.run.smithi085.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-07T17:26:33.185 INFO:teuthology.orchestra.run.smithi085.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-07T17:26:33.264 INFO:teuthology.orchestra.run.smithi057.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-07T17:26:33.314 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T17:26:33.330 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:33.654 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:34.000 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T17:26:34.001 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T17:26:34.268 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:34.632 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:35.205 INFO:teuthology.orchestra.run.smithi066.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-07T17:26:35.206 INFO:teuthology.orchestra.run.smithi066.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-07T17:26:35.371 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:35.639 INFO:teuthology.orchestra.run.smithi057.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-07T17:26:35.639 INFO:teuthology.orchestra.run.smithi057.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-07T17:26:35.820 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:35.975 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T17:26:36.126 INFO:teuthology.orchestra.run.smithi085.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-07T17:26:36.740 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:37.163 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:38.149 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T17:26:38.275 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-07T17:26:38.548 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T17:26:38.682 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-07T17:26:40.798 INFO:teuthology.orchestra.run.smithi085.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T17:26:41.484 INFO:teuthology.orchestra.run.smithi085.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-07T17:26:41.584 INFO:teuthology.orchestra.run.smithi085.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:41.772 INFO:teuthology.orchestra.run.smithi085.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-07T17:26:41.773 INFO:teuthology.orchestra.run.smithi085.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:26:42.996 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T17:26:42.997 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T17:26:43.650 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-07T17:26:43.731 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-07T17:26:43.758 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:43.865 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T17:26:43.975 INFO:teuthology.orchestra.run.smithi066.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-07T17:26:43.975 INFO:teuthology.orchestra.run.smithi066.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:26:44.053 INFO:teuthology.orchestra.run.smithi057.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-07T17:26:44.053 INFO:teuthology.orchestra.run.smithi057.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:05.201 INFO:teuthology.orchestra.run.smithi085.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-07T17:27:05.201 INFO:teuthology.orchestra.run.smithi085.stdout:Sourcing file `/etc/default/grub' 2022-03-07T17:27:05.213 INFO:teuthology.orchestra.run.smithi085.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T17:27:05.229 INFO:teuthology.orchestra.run.smithi085.stdout:Generating grub configuration file ... 2022-03-07T17:27:05.511 INFO:teuthology.orchestra.run.smithi085.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T17:27:05.537 INFO:teuthology.orchestra.run.smithi085.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:05.912 INFO:teuthology.orchestra.run.smithi085.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T17:27:05.924 INFO:teuthology.orchestra.run.smithi085.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T17:27:07.647 INFO:teuthology.orchestra.run.smithi066.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-07T17:27:07.647 INFO:teuthology.orchestra.run.smithi066.stdout:Sourcing file `/etc/default/grub' 2022-03-07T17:27:07.652 INFO:teuthology.orchestra.run.smithi066.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T17:27:07.670 INFO:teuthology.orchestra.run.smithi085.stdout:done 2022-03-07T17:27:07.684 INFO:teuthology.orchestra.run.smithi066.stdout:Generating grub configuration file ... 2022-03-07T17:27:07.824 INFO:teuthology.orchestra.run.smithi057.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-07T17:27:07.825 INFO:teuthology.orchestra.run.smithi057.stdout:Sourcing file `/etc/default/grub' 2022-03-07T17:27:07.829 INFO:teuthology.orchestra.run.smithi057.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T17:27:07.853 INFO:teuthology.orchestra.run.smithi057.stdout:Generating grub configuration file ... 2022-03-07T17:27:08.027 DEBUG:teuthology.orchestra.run.smithi085:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T17:27:08.056 INFO:teuthology.orchestra.run.smithi085.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T17:27:08.056 INFO:teuthology.orchestra.run.smithi085.stdout:Status: install ok installed 2022-03-07T17:27:08.056 INFO:teuthology.orchestra.run.smithi085.stdout:Priority: optional 2022-03-07T17:27:08.056 INFO:teuthology.orchestra.run.smithi085.stdout:Section: kernel 2022-03-07T17:27:08.057 INFO:teuthology.orchestra.run.smithi085.stdout:Installed-Size: 19 2022-03-07T17:27:08.057 INFO:teuthology.orchestra.run.smithi085.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T17:27:08.057 INFO:teuthology.orchestra.run.smithi085.stdout:Architecture: amd64 2022-03-07T17:27:08.057 INFO:teuthology.orchestra.run.smithi085.stdout:Source: linux-meta-hwe-5.13 2022-03-07T17:27:08.058 INFO:teuthology.orchestra.run.smithi085.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T17:27:08.058 INFO:teuthology.orchestra.run.smithi085.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T17:27:08.058 INFO:teuthology.orchestra.run.smithi085.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T17:27:08.058 INFO:teuthology.orchestra.run.smithi085.stdout:Recommends: thermald 2022-03-07T17:27:08.058 INFO:teuthology.orchestra.run.smithi085.stdout:Description: Generic Linux kernel image 2022-03-07T17:27:08.059 INFO:teuthology.orchestra.run.smithi085.stdout: This package will always depend on the latest generic kernel image 2022-03-07T17:27:08.059 INFO:teuthology.orchestra.run.smithi085.stdout: available. 2022-03-07T17:27:08.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-07T17:27:08.059 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-07T17:27:08.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-07T17:27:08.060 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-07T17:27:08.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi085.front.sepia.ceph.com, path=None, version=distro) 2022-03-07T17:27:08.061 DEBUG:teuthology.orchestra.run.smithi085:> sudo apt-get clean 2022-03-07T17:27:08.088 INFO:teuthology.orchestra.run.smithi066.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T17:27:08.119 DEBUG:teuthology.orchestra.run.smithi085:> sudo apt-get update 2022-03-07T17:27:08.123 INFO:teuthology.orchestra.run.smithi066.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:08.175 INFO:teuthology.orchestra.run.smithi057.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T17:27:08.231 INFO:teuthology.orchestra.run.smithi057.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:08.384 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T17:27:08.384 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:27:08.455 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T17:27:08.537 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T17:27:08.556 INFO:teuthology.orchestra.run.smithi066.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T17:27:08.568 INFO:teuthology.orchestra.run.smithi066.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T17:27:08.619 INFO:teuthology.orchestra.run.smithi057.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T17:27:08.630 INFO:teuthology.orchestra.run.smithi057.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T17:27:09.913 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:27:09.935 DEBUG:teuthology.orchestra.run.smithi085:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T17:27:09.998 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:27:10.170 INFO:teuthology.orchestra.run.smithi085.stdout:Building dependency tree... 2022-03-07T17:27:10.171 INFO:teuthology.orchestra.run.smithi085.stdout:Reading state information... 2022-03-07T17:27:10.342 INFO:teuthology.orchestra.run.smithi085.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-07T17:27:10.342 INFO:teuthology.orchestra.run.smithi085.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T17:27:10.343 DEBUG:teuthology.orchestra.run.smithi085:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T17:27:10.359 INFO:teuthology.orchestra.run.smithi085.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T17:27:10.360 INFO:teuthology.orchestra.run.smithi085.stdout:Status: install ok installed 2022-03-07T17:27:10.360 INFO:teuthology.orchestra.run.smithi085.stdout:Priority: optional 2022-03-07T17:27:10.360 INFO:teuthology.orchestra.run.smithi085.stdout:Section: kernel 2022-03-07T17:27:10.360 INFO:teuthology.orchestra.run.smithi085.stdout:Installed-Size: 19 2022-03-07T17:27:10.361 INFO:teuthology.orchestra.run.smithi085.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T17:27:10.361 INFO:teuthology.orchestra.run.smithi085.stdout:Architecture: amd64 2022-03-07T17:27:10.361 INFO:teuthology.orchestra.run.smithi085.stdout:Source: linux-meta-hwe-5.13 2022-03-07T17:27:10.361 INFO:teuthology.orchestra.run.smithi085.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T17:27:10.362 INFO:teuthology.orchestra.run.smithi085.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T17:27:10.362 INFO:teuthology.orchestra.run.smithi085.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T17:27:10.362 INFO:teuthology.orchestra.run.smithi085.stdout:Recommends: thermald 2022-03-07T17:27:10.362 INFO:teuthology.orchestra.run.smithi085.stdout:Description: Generic Linux kernel image 2022-03-07T17:27:10.362 INFO:teuthology.orchestra.run.smithi085.stdout: This package will always depend on the latest generic kernel image 2022-03-07T17:27:10.363 INFO:teuthology.orchestra.run.smithi085.stdout: available. 2022-03-07T17:27:10.363 DEBUG:teuthology.orchestra.run.smithi085:> mktemp 2022-03-07T17:27:10.368 INFO:teuthology.orchestra.run.smithi085.stdout:/tmp/tmp.IpwWyXIKNr 2022-03-07T17:27:10.369 DEBUG:teuthology.orchestra.run.smithi085:> sudo cp /boot/grub/grub.cfg /tmp/tmp.IpwWyXIKNr 2022-03-07T17:27:10.427 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod 0666 /tmp/tmp.IpwWyXIKNr 2022-03-07T17:27:10.473 DEBUG:teuthology.orchestra.remote:smithi085:/tmp/tmp.IpwWyXIKNr is 8KB 2022-03-07T17:27:10.488 DEBUG:teuthology.orchestra.run.smithi085:> rm -fr /tmp/tmp.IpwWyXIKNr 2022-03-07T17:27:10.495 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-07T17:27:10.552 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:27:10.552 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-07T17:27:10.552 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-07T17:27:10.583 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-07T17:27:10.583 DEBUG:teuthology.orchestra.run.smithi085:> sudo update-grub 2022-03-07T17:27:11.243 INFO:teuthology.orchestra.run.smithi066.stdout:done 2022-03-07T17:27:11.320 INFO:teuthology.orchestra.run.smithi057.stdout:done 2022-03-07T17:27:11.607 DEBUG:teuthology.orchestra.run.smithi066:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T17:27:11.639 INFO:teuthology.orchestra.run.smithi066.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T17:27:11.640 INFO:teuthology.orchestra.run.smithi066.stdout:Status: install ok installed 2022-03-07T17:27:11.640 INFO:teuthology.orchestra.run.smithi066.stdout:Priority: optional 2022-03-07T17:27:11.640 INFO:teuthology.orchestra.run.smithi066.stdout:Section: kernel 2022-03-07T17:27:11.640 INFO:teuthology.orchestra.run.smithi066.stdout:Installed-Size: 19 2022-03-07T17:27:11.641 INFO:teuthology.orchestra.run.smithi066.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T17:27:11.641 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: amd64 2022-03-07T17:27:11.641 INFO:teuthology.orchestra.run.smithi066.stdout:Source: linux-meta-hwe-5.13 2022-03-07T17:27:11.641 INFO:teuthology.orchestra.run.smithi066.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T17:27:11.642 INFO:teuthology.orchestra.run.smithi066.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T17:27:11.642 INFO:teuthology.orchestra.run.smithi066.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T17:27:11.642 INFO:teuthology.orchestra.run.smithi066.stdout:Recommends: thermald 2022-03-07T17:27:11.642 INFO:teuthology.orchestra.run.smithi066.stdout:Description: Generic Linux kernel image 2022-03-07T17:27:11.643 INFO:teuthology.orchestra.run.smithi066.stdout: This package will always depend on the latest generic kernel image 2022-03-07T17:27:11.643 INFO:teuthology.orchestra.run.smithi066.stdout: available. 2022-03-07T17:27:11.643 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-07T17:27:11.643 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-07T17:27:11.644 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-07T17:27:11.644 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-07T17:27:11.644 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2022-03-07T17:27:11.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get clean 2022-03-07T17:27:11.706 DEBUG:teuthology.orchestra.run.smithi057:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T17:27:11.708 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get update 2022-03-07T17:27:11.713 INFO:teuthology.orchestra.run.smithi085.stderr:Sourcing file `/etc/default/grub' 2022-03-07T17:27:11.715 INFO:teuthology.orchestra.run.smithi085.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T17:27:11.728 INFO:teuthology.orchestra.run.smithi085.stderr:Generating grub configuration file ... 2022-03-07T17:27:11.738 INFO:teuthology.orchestra.run.smithi057.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T17:27:11.738 INFO:teuthology.orchestra.run.smithi057.stdout:Status: install ok installed 2022-03-07T17:27:11.739 INFO:teuthology.orchestra.run.smithi057.stdout:Priority: optional 2022-03-07T17:27:11.739 INFO:teuthology.orchestra.run.smithi057.stdout:Section: kernel 2022-03-07T17:27:11.739 INFO:teuthology.orchestra.run.smithi057.stdout:Installed-Size: 19 2022-03-07T17:27:11.739 INFO:teuthology.orchestra.run.smithi057.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T17:27:11.740 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: amd64 2022-03-07T17:27:11.740 INFO:teuthology.orchestra.run.smithi057.stdout:Source: linux-meta-hwe-5.13 2022-03-07T17:27:11.740 INFO:teuthology.orchestra.run.smithi057.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T17:27:11.740 INFO:teuthology.orchestra.run.smithi057.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T17:27:11.740 INFO:teuthology.orchestra.run.smithi057.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T17:27:11.741 INFO:teuthology.orchestra.run.smithi057.stdout:Recommends: thermald 2022-03-07T17:27:11.741 INFO:teuthology.orchestra.run.smithi057.stdout:Description: Generic Linux kernel image 2022-03-07T17:27:11.741 INFO:teuthology.orchestra.run.smithi057.stdout: This package will always depend on the latest generic kernel image 2022-03-07T17:27:11.741 INFO:teuthology.orchestra.run.smithi057.stdout: available. 2022-03-07T17:27:11.742 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-07T17:27:11.742 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-07T17:27:11.742 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-07T17:27:11.742 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-07T17:27:11.743 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2022-03-07T17:27:11.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get clean 2022-03-07T17:27:11.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2022-03-07T17:27:11.965 INFO:teuthology.orchestra.run.smithi085.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T17:27:11.972 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:27:11.973 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T17:27:11.977 INFO:teuthology.orchestra.run.smithi085.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:12.043 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T17:27:12.116 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T17:27:12.121 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:27:12.131 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T17:27:12.203 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T17:27:12.285 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T17:27:12.334 INFO:teuthology.orchestra.run.smithi085.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T17:27:12.346 INFO:teuthology.orchestra.run.smithi085.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T17:27:13.054 INFO:teuthology.orchestra.run.smithi085.stderr:done 2022-03-07T17:27:13.056 DEBUG:teuthology.orchestra.run.smithi085:> sudo shutdown -r now 2022-03-07T17:27:13.538 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:27:13.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T17:27:13.626 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:27:13.684 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:27:13.708 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T17:27:13.775 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:27:13.809 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-07T17:27:13.809 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-07T17:27:13.940 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-07T17:27:13.941 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-07T17:27:14.000 INFO:teuthology.orchestra.run.smithi066.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-07T17:27:14.001 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T17:27:14.002 DEBUG:teuthology.orchestra.run.smithi066:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T17:27:14.022 INFO:teuthology.orchestra.run.smithi066.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T17:27:14.023 INFO:teuthology.orchestra.run.smithi066.stdout:Status: install ok installed 2022-03-07T17:27:14.023 INFO:teuthology.orchestra.run.smithi066.stdout:Priority: optional 2022-03-07T17:27:14.023 INFO:teuthology.orchestra.run.smithi066.stdout:Section: kernel 2022-03-07T17:27:14.024 INFO:teuthology.orchestra.run.smithi066.stdout:Installed-Size: 19 2022-03-07T17:27:14.024 INFO:teuthology.orchestra.run.smithi066.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T17:27:14.024 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: amd64 2022-03-07T17:27:14.024 INFO:teuthology.orchestra.run.smithi066.stdout:Source: linux-meta-hwe-5.13 2022-03-07T17:27:14.025 INFO:teuthology.orchestra.run.smithi066.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T17:27:14.025 INFO:teuthology.orchestra.run.smithi066.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T17:27:14.025 INFO:teuthology.orchestra.run.smithi066.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T17:27:14.025 INFO:teuthology.orchestra.run.smithi066.stdout:Recommends: thermald 2022-03-07T17:27:14.026 INFO:teuthology.orchestra.run.smithi066.stdout:Description: Generic Linux kernel image 2022-03-07T17:27:14.026 INFO:teuthology.orchestra.run.smithi066.stdout: This package will always depend on the latest generic kernel image 2022-03-07T17:27:14.026 INFO:teuthology.orchestra.run.smithi066.stdout: available. 2022-03-07T17:27:14.026 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2022-03-07T17:27:14.031 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.NiaU19mj2B 2022-03-07T17:27:14.031 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NiaU19mj2B 2022-03-07T17:27:14.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.NiaU19mj2B 2022-03-07T17:27:14.117 INFO:teuthology.orchestra.run.smithi057.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-07T17:27:14.117 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T17:27:14.119 DEBUG:teuthology.orchestra.run.smithi057:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T17:27:14.128 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.NiaU19mj2B is 8KB 2022-03-07T17:27:14.138 INFO:teuthology.orchestra.run.smithi057.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T17:27:14.139 INFO:teuthology.orchestra.run.smithi057.stdout:Status: install ok installed 2022-03-07T17:27:14.139 INFO:teuthology.orchestra.run.smithi057.stdout:Priority: optional 2022-03-07T17:27:14.139 INFO:teuthology.orchestra.run.smithi057.stdout:Section: kernel 2022-03-07T17:27:14.139 INFO:teuthology.orchestra.run.smithi057.stdout:Installed-Size: 19 2022-03-07T17:27:14.140 INFO:teuthology.orchestra.run.smithi057.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T17:27:14.140 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: amd64 2022-03-07T17:27:14.140 INFO:teuthology.orchestra.run.smithi057.stdout:Source: linux-meta-hwe-5.13 2022-03-07T17:27:14.140 INFO:teuthology.orchestra.run.smithi057.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T17:27:14.141 INFO:teuthology.orchestra.run.smithi057.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T17:27:14.141 INFO:teuthology.orchestra.run.smithi057.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T17:27:14.141 INFO:teuthology.orchestra.run.smithi057.stdout:Recommends: thermald 2022-03-07T17:27:14.141 INFO:teuthology.orchestra.run.smithi057.stdout:Description: Generic Linux kernel image 2022-03-07T17:27:14.141 INFO:teuthology.orchestra.run.smithi057.stdout: This package will always depend on the latest generic kernel image 2022-03-07T17:27:14.142 INFO:teuthology.orchestra.run.smithi057.stdout: available. 2022-03-07T17:27:14.142 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2022-03-07T17:27:14.147 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.NiaU19mj2B 2022-03-07T17:27:14.154 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-07T17:27:14.188 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.Gvn4qx92p4 2022-03-07T17:27:14.188 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Gvn4qx92p4 2022-03-07T17:27:14.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.Gvn4qx92p4 2022-03-07T17:27:14.209 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:27:14.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-07T17:27:14.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-07T17:27:14.241 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.Gvn4qx92p4 is 8KB 2022-03-07T17:27:14.254 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.Gvn4qx92p4 2022-03-07T17:27:14.258 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-07T17:27:14.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo update-grub 2022-03-07T17:27:14.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-07T17:27:14.315 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:27:14.315 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-07T17:27:14.316 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-07T17:27:14.370 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-07T17:27:14.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo update-grub 2022-03-07T17:27:15.458 INFO:teuthology.orchestra.run.smithi066.stderr:Sourcing file `/etc/default/grub' 2022-03-07T17:27:15.460 INFO:teuthology.orchestra.run.smithi066.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T17:27:15.466 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2022-03-07T17:27:15.685 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub' 2022-03-07T17:27:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T17:27:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2022-03-07T17:27:15.696 INFO:teuthology.orchestra.run.smithi066.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T17:27:15.708 INFO:teuthology.orchestra.run.smithi066.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:15.946 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T17:27:15.957 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T17:27:16.043 INFO:teuthology.orchestra.run.smithi066.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T17:27:16.054 INFO:teuthology.orchestra.run.smithi066.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T17:27:16.363 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T17:27:16.374 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T17:27:17.632 INFO:teuthology.orchestra.run.smithi066.stderr:done 2022-03-07T17:27:17.633 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2022-03-07T17:27:18.019 INFO:teuthology.orchestra.run.smithi057.stderr:done 2022-03-07T17:27:18.021 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2022-03-07T17:27:43.083 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-07T17:27:43.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-03-07T17:27:43.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:27:47.641 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-07T17:27:47.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-07T17:27:47.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:27:48.025 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-07T17:27:48.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-07T17:27:48.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:28:01.483 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.85 2022-03-07T17:28:06.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2022-03-07T17:28:06.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2022-03-07T17:28:31.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-03-07T17:28:31.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:28:36.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-07T17:28:36.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:28:36.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-07T17:28:36.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:28:39.179 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2022-03-07T17:28:47.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.85 2022-03-07T17:28:54.731 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2022-03-07T17:29:09.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-07T17:29:09.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:29:12.239 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2022-03-07T17:29:17.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-03-07T17:29:17.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:29:18.012 DEBUG:teuthology.orchestra.run.smithi085:> true 2022-03-07T17:29:18.520 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-03-07T17:29:18.521 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-03-07T17:29:18.521 DEBUG:teuthology.orchestra.run.smithi085:> uname -r 2022-03-07T17:29:18.526 INFO:teuthology.orchestra.run.smithi085.stdout:5.13.0-30-generic 2022-03-07T17:29:18.526 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-07T17:29:18.527 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-07T17:29:18.527 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-07T17:29:19.528 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-07T17:29:19.529 DEBUG:teuthology.orchestra.run.smithi085:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-07T17:29:19.886 INFO:teuthology.orchestra.run.smithi085.stdout:ttyS1 2022-03-07T17:29:19.911 DEBUG:teuthology.parallel:result is None 2022-03-07T17:29:24.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-07T17:29:24.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:29:24.976 DEBUG:teuthology.orchestra.run.smithi057:> true 2022-03-07T17:29:25.450 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-07T17:29:25.450 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-07T17:29:25.451 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2022-03-07T17:29:25.456 INFO:teuthology.orchestra.run.smithi057.stdout:5.13.0-30-generic 2022-03-07T17:29:25.456 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-07T17:29:25.457 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-07T17:29:25.457 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-07T17:29:26.458 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-07T17:29:26.458 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-07T17:29:26.526 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2022-03-07T17:29:26.560 DEBUG:teuthology.parallel:result is None 2022-03-07T17:29:42.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-07T17:29:42.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:29:42.381 DEBUG:teuthology.orchestra.run.smithi066:> true 2022-03-07T17:29:42.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-07T17:29:42.914 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-03-07T17:29:42.914 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2022-03-07T17:29:42.922 INFO:teuthology.orchestra.run.smithi066.stdout:5.13.0-30-generic 2022-03-07T17:29:42.922 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-07T17:29:42.922 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-07T17:29:42.923 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-07T17:29:43.924 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-07T17:29:43.924 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-07T17:29:43.993 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2022-03-07T17:29:44.024 DEBUG:teuthology.parallel:result is None 2022-03-07T17:29:44.025 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-07T17:29:44.036 INFO:teuthology.task.internal:Creating test directory... 2022-03-07T17:29:44.036 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-07T17:29:44.039 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-07T17:29:44.042 DEBUG:teuthology.orchestra.run.smithi085:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-07T17:29:44.050 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-07T17:29:44.071 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-07T17:29:44.104 INFO:teuthology.task.internal:Creating archive directory... 2022-03-07T17:29:44.105 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-07T17:29:44.107 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-07T17:29:44.110 DEBUG:teuthology.orchestra.run.smithi085:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-07T17:29:44.201 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-07T17:29:44.211 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-07T17:29:44.212 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-07T17:29:44.215 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-07T17:29:44.217 DEBUG:teuthology.orchestra.run.smithi085:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-07T17:29:44.236 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T17:29:44.237 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T17:29:44.242 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T17:29:44.244 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T17:29:44.244 INFO:teuthology.orchestra.run.smithi085.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T17:29:44.253 INFO:teuthology.orchestra.run.smithi085.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T17:29:44.254 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-07T17:29:44.265 INFO:teuthology.task.internal:Configuring sudo... 2022-03-07T17:29:44.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-07T17:29:44.268 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-07T17:29:44.288 DEBUG:teuthology.orchestra.run.smithi085:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-07T17:29:44.313 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-07T17:29:44.324 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-07T17:29:44.324 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-07T17:29:44.327 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-07T17:29:44.330 DEBUG:teuthology.orchestra.run.smithi085:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-07T17:29:44.337 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-07T17:29:44.344 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-07T17:29:44.351 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:29:44.351 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-07T17:29:44.411 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-07T17:29:44.419 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-07T17:29:44.427 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:29:44.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-07T17:29:44.452 DEBUG:teuthology.orchestra.run.smithi085:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-07T17:29:44.460 DEBUG:teuthology.orchestra.run.smithi085:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-07T17:29:44.466 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:29:44.467 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-07T17:29:44.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2022-03-07T17:29:44.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2022-03-07T17:29:44.496 DEBUG:teuthology.orchestra.run.smithi085:> sudo service rsyslog restart 2022-03-07T17:29:44.577 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-07T17:29:44.588 INFO:teuthology.task.internal:Starting timer... 2022-03-07T17:29:44.588 INFO:teuthology.run_tasks:Running task pcp... 2022-03-07T17:29:44.632 INFO:teuthology.run_tasks:Running task selinux... 2022-03-07T17:29:44.665 DEBUG:teuthology.task.selinux:Excluding smithi057: OS 'ubuntu' does not support SELinux 2022-03-07T17:29:44.665 DEBUG:teuthology.task.selinux:Excluding smithi066: OS 'ubuntu' does not support SELinux 2022-03-07T17:29:44.666 DEBUG:teuthology.task.selinux:Excluding smithi085: OS 'ubuntu' does not support SELinux 2022-03-07T17:29:44.666 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-07T17:29:44.666 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-07T17:29:44.667 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-07T17:29:44.667 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-07T17:29:44.700 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-07T17:29:44.706 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-07T17:29:44.740 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-07T17:29:44.760 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-07T17:29:44.762 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi066.front.sepia.ceph.com,smithi085.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-07T17:32:50.238 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com'), Remote(name='ubuntu@smithi085.front.sepia.ceph.com')] 2022-03-07T17:32:50.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-07T17:32:50.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:32:50.294 DEBUG:teuthology.orchestra.run.smithi057:> true 2022-03-07T17:32:50.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-07T17:32:50.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:32:50.465 DEBUG:teuthology.orchestra.run.smithi066:> true 2022-03-07T17:32:50.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-03-07T17:32:50.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T17:32:50.634 DEBUG:teuthology.orchestra.run.smithi085:> true 2022-03-07T17:32:50.749 INFO:teuthology.run_tasks:Running task clock... 2022-03-07T17:32:50.762 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-07T17:32:50.762 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-07T17:32:50.764 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-07T17:32:50.767 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-07T17:32:50.767 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-07T17:32:50.770 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-07T17:32:50.770 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-07T17:32:50.799 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-07T17:32:50.799 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Command line: ntpd -gq 2022-03-07T17:32:50.800 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: proto: precision = 0.074 usec (-24) 2022-03-07T17:32:50.800 INFO:teuthology.orchestra.run.smithi057.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T17:32:50.801 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T17:32:50.801 INFO:teuthology.orchestra.run.smithi057.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-07T17:32:50.802 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: restrict ::: KOD does nothing without LIMITED. 2022-03-07T17:32:50.803 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listen and drop on 0 v6wildcard [::]:123 2022-03-07T17:32:50.803 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-07T17:32:50.803 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listen normally on 2 lo 127.0.0.1:123 2022-03-07T17:32:50.803 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listen normally on 3 ens1f0 172.21.15.57:123 2022-03-07T17:32:50.803 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listen normally on 4 lo [::1]:123 2022-03-07T17:32:50.804 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1562%3]:123 2022-03-07T17:32:50.804 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:50 ntpd[8901]: Listening on routing socket on fd #22 for interface updates 2022-03-07T17:32:50.805 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-07T17:32:50.805 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Command line: ntpd -gq 2022-03-07T17:32:50.805 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-07T17:32:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Command line: ntpd -gq 2022-03-07T17:32:50.806 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: proto: precision = 0.114 usec (-23) 2022-03-07T17:32:50.806 INFO:teuthology.orchestra.run.smithi066.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T17:32:50.807 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T17:32:50.807 INFO:teuthology.orchestra.run.smithi066.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-07T17:32:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: restrict ::: KOD does nothing without LIMITED. 2022-03-07T17:32:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listen and drop on 0 v6wildcard [::]:123 2022-03-07T17:32:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-07T17:32:50.809 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listen normally on 2 lo 127.0.0.1:123 2022-03-07T17:32:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listen normally on 3 enp3s0f1 172.21.15.66:123 2022-03-07T17:32:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listen normally on 4 lo [::1]:123 2022-03-07T17:32:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8bd%5]:123 2022-03-07T17:32:50.810 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:50 ntpd[9009]: Listening on routing socket on fd #22 for interface updates 2022-03-07T17:32:50.811 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: proto: precision = 0.100 usec (-23) 2022-03-07T17:32:50.811 INFO:teuthology.orchestra.run.smithi085.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T17:32:50.812 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T17:32:50.812 INFO:teuthology.orchestra.run.smithi085.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-07T17:32:50.814 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: restrict ::: KOD does nothing without LIMITED. 2022-03-07T17:32:50.814 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listen and drop on 0 v6wildcard [::]:123 2022-03-07T17:32:50.814 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-07T17:32:50.814 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listen normally on 2 lo 127.0.0.1:123 2022-03-07T17:32:50.815 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listen normally on 3 enp3s0f1 172.21.15.85:123 2022-03-07T17:32:50.815 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listen normally on 4 lo [::1]:123 2022-03-07T17:32:50.815 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d7bb%5]:123 2022-03-07T17:32:50.815 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:50 ntpd[9008]: Listening on routing socket on fd #22 for interface updates 2022-03-07T17:32:51.797 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:51.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:51.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:51.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:51 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:51.803 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:51 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.803 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:51 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:51 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:51 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:51 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:51.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:51 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:51.808 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.808 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.808 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.809 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.809 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.809 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.810 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.810 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.810 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:51.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:51.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:51.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:51.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:51.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:51.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:51.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:51 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.797 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.797 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:53 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.804 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:53 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:53.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:53.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:53.817 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:53.817 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:53 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:54.804 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:54.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:54.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:54.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:54.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:54.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:54.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:54.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:54.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:54.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:54.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:54.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:54 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.798 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.803 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.803 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.803 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.803 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.804 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.804 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:55 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.808 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.808 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.808 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.808 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.809 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.809 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.809 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.809 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.809 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:55 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.814 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:55.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:55.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:55.816 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:55.817 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:55 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:56.804 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:56.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:56.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:56.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:56.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:56.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:56.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:56.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:56.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:56.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:56.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:56.808 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:56 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:57.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.799 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.800 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:57.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:57.801 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:57.802 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:57.803 INFO:teuthology.orchestra.run.smithi057.stdout: 7 Mar 17:32:57 ntpd[8901]: ntpd: time slew -0.001260 s 2022-03-07T17:32:57.803 INFO:teuthology.orchestra.run.smithi057.stdout:ntpd: time slew -0.001260s 2022-03-07T17:32:57.803 INFO:teuthology.orchestra.run.smithi057.stderr: 7 Mar 17:32:57 ntpd[8901]: can't open /var/log/ntpstats/loopstats.20220307: Permission denied 2022-03-07T17:32:57.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.805 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.806 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:57.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:57.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.807 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:57.808 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:57.808 INFO:teuthology.orchestra.run.smithi066.stdout: 7 Mar 17:32:57 ntpd[9009]: ntpd: time slew -0.000250 s 2022-03-07T17:32:57.808 INFO:teuthology.orchestra.run.smithi066.stdout:ntpd: time slew -0.000250s 2022-03-07T17:32:57.809 INFO:teuthology.orchestra.run.smithi066.stderr: 7 Mar 17:32:57 ntpd[9009]: can't open /var/log/ntpstats/loopstats.20220307: Permission denied 2022-03-07T17:32:57.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.811 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.812 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T17:32:57.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T17:32:57.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T17:32:57.813 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T17:32:57.814 INFO:teuthology.orchestra.run.smithi085.stdout: 7 Mar 17:32:57 ntpd[9008]: ntpd: time slew -0.000974 s 2022-03-07T17:32:57.814 INFO:teuthology.orchestra.run.smithi085.stdout:ntpd: time slew -0.000974s 2022-03-07T17:32:57.815 INFO:teuthology.orchestra.run.smithi085.stderr: 7 Mar 17:32:57 ntpd[9008]: can't open /var/log/ntpstats/loopstats.20220307: Permission denied 2022-03-07T17:32:57.852 INFO:teuthology.orchestra.run.smithi057.stdout: remote refid st t when poll reach delay offset jitter 2022-03-07T17:32:57.852 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================== 2022-03-07T17:32:57.853 INFO:teuthology.orchestra.run.smithi057.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.853 INFO:teuthology.orchestra.run.smithi057.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.853 INFO:teuthology.orchestra.run.smithi057.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.853 INFO:teuthology.orchestra.run.smithi057.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.856 INFO:teuthology.orchestra.run.smithi066.stdout: remote refid st t when poll reach delay offset jitter 2022-03-07T17:32:57.856 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================== 2022-03-07T17:32:57.856 INFO:teuthology.orchestra.run.smithi066.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.857 INFO:teuthology.orchestra.run.smithi066.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.857 INFO:teuthology.orchestra.run.smithi066.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.857 INFO:teuthology.orchestra.run.smithi066.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.921 INFO:teuthology.orchestra.run.smithi085.stdout: remote refid st t when poll reach delay offset jitter 2022-03-07T17:32:57.922 INFO:teuthology.orchestra.run.smithi085.stdout:============================================================================== 2022-03-07T17:32:57.922 INFO:teuthology.orchestra.run.smithi085.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.922 INFO:teuthology.orchestra.run.smithi085.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.922 INFO:teuthology.orchestra.run.smithi085.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.923 INFO:teuthology.orchestra.run.smithi085.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T17:32:57.924 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-07T17:32:57.940 INFO:tasks.kubeadm:Kubeadm start 2022-03-07T17:32:57.940 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-07T17:32:57.941 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-07T17:32:57.944 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-07T17:32:57.946 DEBUG:teuthology.orchestra.run.smithi085:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-07T17:32:57.969 INFO:teuthology.orchestra.run.smithi066.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-07T17:32:57.969 INFO:teuthology.orchestra.run.smithi057.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-07T17:32:57.977 INFO:teuthology.orchestra.run.smithi066.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-07T17:32:57.980 INFO:teuthology.orchestra.run.smithi057.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-07T17:32:57.985 INFO:teuthology.orchestra.run.smithi066.stdout:net.ipv4.ip_forward = 1 2022-03-07T17:32:57.989 INFO:teuthology.orchestra.run.smithi085.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-07T17:32:57.990 INFO:teuthology.orchestra.run.smithi057.stdout:net.ipv4.ip_forward = 1 2022-03-07T17:32:57.996 INFO:teuthology.orchestra.run.smithi085.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-07T17:32:58.003 INFO:teuthology.orchestra.run.smithi085.stdout:net.ipv4.ip_forward = 1 2022-03-07T17:32:58.052 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:32:58.053 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-07T17:32:58.068 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:32:58.069 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/docker/daemon.json 2022-03-07T17:32:58.085 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:32:58.085 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-07T17:32:58.108 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:32:58.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/docker/daemon.json 2022-03-07T17:32:58.120 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:32:58.120 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-07T17:32:58.137 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:32:58.137 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/docker/daemon.json 2022-03-07T17:32:58.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl restart docker || true 2022-03-07T17:32:58.152 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl restart docker || true 2022-03-07T17:32:58.155 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl restart docker || true 2022-03-07T17:32:59.166 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-07T17:32:59.177 INFO:tasks.kubeadm:172.21.15.57 is in 172.21.0.0/20 at pos 3896 2022-03-07T17:32:59.203 INFO:tasks.kubeadm:Pod subnet: 10.249.192.0/21 2022-03-07T17:32:59.204 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-07T17:32:59.206 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-07T17:32:59.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-07T17:32:59.213 DEBUG:teuthology.orchestra.run.smithi085:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-07T17:32:59.231 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:32:59.232 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:32:59.232 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:32:59.235 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:32:59.235 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:32:59.235 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:32:59.240 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:32:59.240 INFO:teuthology.orchestra.run.smithi085.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:32:59.241 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:32:59.337 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:32:59.461 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:32:59.476 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:32:59.477 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:32:59.480 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:32:59.480 INFO:teuthology.orchestra.run.smithi085.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:32:59.546 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:32:59.549 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:32:59.554 INFO:teuthology.orchestra.run.smithi085.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:32:59.906 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:32:59.916 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:32:59.917 INFO:teuthology.orchestra.run.smithi085.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:33:00.179 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 336 kB in 1s (411 kB/s) 2022-03-07T17:33:00.224 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 336 kB in 1s (394 kB/s) 2022-03-07T17:33:00.247 INFO:teuthology.orchestra.run.smithi085.stdout:Fetched 336 kB in 1s (385 kB/s) 2022-03-07T17:33:01.047 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:33:01.101 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:33:01.105 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:33:01.222 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-07T17:33:01.222 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-07T17:33:01.271 INFO:teuthology.orchestra.run.smithi085.stdout:Building dependency tree... 2022-03-07T17:33:01.272 INFO:teuthology.orchestra.run.smithi085.stdout:Reading state information... 2022-03-07T17:33:01.283 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-07T17:33:01.283 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-07T17:33:01.316 INFO:teuthology.orchestra.run.smithi057.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T17:33:01.330 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:33:01.330 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:01.331 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:33:01.367 INFO:teuthology.orchestra.run.smithi085.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T17:33:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T17:33:01.384 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:33:01.385 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:33:01.385 INFO:teuthology.orchestra.run.smithi085.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:01.385 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:33:01.389 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:33:01.389 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:01.390 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:33:01.439 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:33:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:33:01.545 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-07T17:33:01.546 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-07T17:33:01.611 INFO:teuthology.orchestra.run.smithi085.stdout:Building dependency tree... 2022-03-07T17:33:01.612 INFO:teuthology.orchestra.run.smithi085.stdout:Reading state information... 2022-03-07T17:33:01.620 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-07T17:33:01.621 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-07T17:33:01.717 INFO:teuthology.orchestra.run.smithi057.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-07T17:33:01.717 INFO:teuthology.orchestra.run.smithi057.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-07T17:33:01.717 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2022-03-07T17:33:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: apt-transport-https 2022-03-07T17:33:01.793 INFO:teuthology.orchestra.run.smithi085.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-07T17:33:01.794 INFO:teuthology.orchestra.run.smithi085.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-07T17:33:01.794 INFO:teuthology.orchestra.run.smithi085.stdout:The following NEW packages will be installed: 2022-03-07T17:33:01.795 INFO:teuthology.orchestra.run.smithi085.stdout: apt-transport-https 2022-03-07T17:33:01.809 INFO:teuthology.orchestra.run.smithi066.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-07T17:33:01.809 INFO:teuthology.orchestra.run.smithi066.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-07T17:33:01.810 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2022-03-07T17:33:01.811 INFO:teuthology.orchestra.run.smithi066.stdout: apt-transport-https 2022-03-07T17:33:01.902 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T17:33:01.902 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 4,680 B of archives. 2022-03-07T17:33:01.903 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-07T17:33:01.903 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-07T17:33:01.991 INFO:teuthology.orchestra.run.smithi085.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T17:33:01.991 INFO:teuthology.orchestra.run.smithi085.stdout:Need to get 4,680 B of archives. 2022-03-07T17:33:01.992 INFO:teuthology.orchestra.run.smithi085.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-07T17:33:01.992 INFO:teuthology.orchestra.run.smithi085.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-07T17:33:01.996 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T17:33:01.997 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 4,680 B of archives. 2022-03-07T17:33:01.997 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-07T17:33:01.998 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-07T17:33:02.133 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T17:33:02.134 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T17:33:02.134 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Readline 2022-03-07T17:33:02.155 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T17:33:02.155 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T17:33:02.156 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Teletype 2022-03-07T17:33:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T17:33:02.216 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 4,680 B in 0s (26.5 kB/s) 2022-03-07T17:33:02.221 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T17:33:02.221 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T17:33:02.221 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: falling back to frontend: Readline 2022-03-07T17:33:02.237 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T17:33:02.237 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T17:33:02.238 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Readline 2022-03-07T17:33:02.245 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T17:33:02.245 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T17:33:02.245 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: falling back to frontend: Teletype 2022-03-07T17:33:02.250 INFO:teuthology.orchestra.run.smithi085.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T17:33:02.254 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T17:33:02.255 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T17:33:02.255 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Teletype 2022-03-07T17:33:02.261 INFO:teuthology.orchestra.run.smithi066.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T17:33:02.301 INFO:teuthology.orchestra.run.smithi085.stdout:Fetched 4,680 B in 0s (24.7 kB/s) 2022-03-07T17:33:02.306 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package apt-transport-https. 2022-03-07T17:33:02.311 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-07T17:33:02.351 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-07T17:33:02.355 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-07T17:33:02.388 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package apt-transport-https. 2022-03-07T17:33:02.396 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-07T17:33:02.405 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package apt-transport-https. 2022-03-07T17:33:02.430 INFO:teuthology.orchestra.run.smithi085.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-07T17:33:02.433 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-07T17:33:02.449 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-07T17:33:02.453 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-07T17:33:02.478 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-07T17:33:02.495 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-07T17:33:02.691 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-07T17:33:02.759 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-07T17:33:02.784 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-07T17:33:03.218 INFO:teuthology.orchestra.run.smithi057.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-07T17:33:03.234 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:33:03.234 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:03.234 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:33:03.249 INFO:teuthology.orchestra.run.smithi066.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-07T17:33:03.264 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:33:03.264 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:03.264 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:33:03.311 INFO:teuthology.orchestra.run.smithi085.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-07T17:33:03.324 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:33:03.324 INFO:teuthology.orchestra.run.smithi085.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:03.325 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:33:03.389 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:33:03.443 INFO:teuthology.orchestra.run.smithi085.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:33:03.490 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:33:03.491 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T17:33:03.511 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:33:03.564 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:33:03.569 INFO:teuthology.orchestra.run.smithi085.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T17:33:03.594 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:33:03.656 INFO:teuthology.orchestra.run.smithi085.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T17:33:03.905 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:33:03.932 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:33:03.996 INFO:teuthology.orchestra.run.smithi085.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T17:33:04.763 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-07T17:33:04.861 INFO:teuthology.orchestra.run.smithi085.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-07T17:33:04.861 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-07T17:33:04.930 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-07T17:33:04.971 INFO:teuthology.orchestra.run.smithi085.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-07T17:33:05.003 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 399 kB in 2s (252 kB/s) 2022-03-07T17:33:05.025 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-07T17:33:05.113 INFO:teuthology.orchestra.run.smithi085.stdout:Fetched 399 kB in 2s (243 kB/s) 2022-03-07T17:33:05.161 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 399 kB in 2s (225 kB/s) 2022-03-07T17:33:05.877 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:33:05.949 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:33:06.036 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:33:06.052 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-07T17:33:06.052 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-07T17:33:06.122 INFO:teuthology.orchestra.run.smithi085.stdout:Building dependency tree... 2022-03-07T17:33:06.123 INFO:teuthology.orchestra.run.smithi085.stdout:Reading state information... 2022-03-07T17:33:06.145 INFO:teuthology.orchestra.run.smithi066.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T17:33:06.157 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:33:06.157 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:06.157 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-07T17:33:06.212 INFO:teuthology.orchestra.run.smithi085.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T17:33:06.213 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-07T17:33:06.214 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-07T17:33:06.214 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-07T17:33:06.226 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:33:06.226 INFO:teuthology.orchestra.run.smithi085.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:06.227 INFO:teuthology.orchestra.run.smithi085.stderr: 2022-03-07T17:33:06.276 INFO:teuthology.orchestra.run.smithi085.stdout:Reading package lists... 2022-03-07T17:33:06.308 INFO:teuthology.orchestra.run.smithi057.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T17:33:06.325 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:33:06.325 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T17:33:06.325 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:33:06.382 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-07T17:33:06.391 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-07T17:33:06.391 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-07T17:33:06.450 INFO:teuthology.orchestra.run.smithi085.stdout:Building dependency tree... 2022-03-07T17:33:06.451 INFO:teuthology.orchestra.run.smithi085.stdout:Reading state information... 2022-03-07T17:33:06.543 INFO:teuthology.orchestra.run.smithi066.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-07T17:33:06.543 INFO:teuthology.orchestra.run.smithi066.stdout:bridge-utils set to manually installed. 2022-03-07T17:33:06.544 INFO:teuthology.orchestra.run.smithi066.stdout:The following additional packages will be installed: 2022-03-07T17:33:06.544 INFO:teuthology.orchestra.run.smithi066.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-07T17:33:06.545 INFO:teuthology.orchestra.run.smithi066.stdout:Suggested packages: 2022-03-07T17:33:06.545 INFO:teuthology.orchestra.run.smithi066.stdout: nftables 2022-03-07T17:33:06.559 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-07T17:33:06.559 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-07T17:33:06.583 INFO:teuthology.orchestra.run.smithi085.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-07T17:33:06.583 INFO:teuthology.orchestra.run.smithi085.stdout:bridge-utils set to manually installed. 2022-03-07T17:33:06.584 INFO:teuthology.orchestra.run.smithi085.stdout:The following additional packages will be installed: 2022-03-07T17:33:06.584 INFO:teuthology.orchestra.run.smithi085.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-07T17:33:06.585 INFO:teuthology.orchestra.run.smithi085.stdout:Suggested packages: 2022-03-07T17:33:06.585 INFO:teuthology.orchestra.run.smithi085.stdout: nftables 2022-03-07T17:33:06.589 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2022-03-07T17:33:06.590 INFO:teuthology.orchestra.run.smithi066.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-07T17:33:06.632 INFO:teuthology.orchestra.run.smithi085.stdout:The following NEW packages will be installed: 2022-03-07T17:33:06.633 INFO:teuthology.orchestra.run.smithi085.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-07T17:33:06.715 INFO:teuthology.orchestra.run.smithi057.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-07T17:33:06.716 INFO:teuthology.orchestra.run.smithi057.stdout:bridge-utils set to manually installed. 2022-03-07T17:33:06.716 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2022-03-07T17:33:06.716 INFO:teuthology.orchestra.run.smithi057.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-07T17:33:06.717 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2022-03-07T17:33:06.717 INFO:teuthology.orchestra.run.smithi057.stdout: nftables 2022-03-07T17:33:06.767 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2022-03-07T17:33:06.769 INFO:teuthology.orchestra.run.smithi057.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-07T17:33:06.785 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T17:33:06.785 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 73.6 MB of archives. 2022-03-07T17:33:06.786 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-07T17:33:06.786 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-07T17:33:06.829 INFO:teuthology.orchestra.run.smithi085.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T17:33:06.829 INFO:teuthology.orchestra.run.smithi085.stdout:Need to get 73.6 MB of archives. 2022-03-07T17:33:06.830 INFO:teuthology.orchestra.run.smithi085.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-07T17:33:06.830 INFO:teuthology.orchestra.run.smithi085.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-07T17:33:06.866 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-07T17:33:06.904 INFO:teuthology.orchestra.run.smithi085.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-07T17:33:06.968 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-07T17:33:06.977 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T17:33:06.977 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 73.6 MB of archives. 2022-03-07T17:33:06.977 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-07T17:33:06.977 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-07T17:33:07.006 INFO:teuthology.orchestra.run.smithi085.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-07T17:33:07.053 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-07T17:33:07.124 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-07T17:33:07.180 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-07T17:33:07.277 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-07T17:33:07.321 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-07T17:33:07.359 INFO:teuthology.orchestra.run.smithi085.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-07T17:33:07.478 INFO:teuthology.orchestra.run.smithi085.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-07T17:33:07.534 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-07T17:33:08.210 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-07T17:33:08.323 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-07T17:33:08.389 INFO:teuthology.orchestra.run.smithi085.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-07T17:33:08.692 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-07T17:33:08.842 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-07T17:33:09.113 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-07T17:33:09.246 INFO:teuthology.orchestra.run.smithi085.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-07T17:33:09.254 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T17:33:09.254 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T17:33:09.254 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Readline 2022-03-07T17:33:09.263 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T17:33:09.263 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T17:33:09.264 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Teletype 2022-03-07T17:33:09.268 INFO:teuthology.orchestra.run.smithi057.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T17:33:09.322 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 73.6 MB in 2s (32.7 MB/s) 2022-03-07T17:33:09.335 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-07T17:33:09.414 INFO:teuthology.orchestra.run.smithi085.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-07T17:33:09.519 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package conntrack. 2022-03-07T17:33:09.565 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-07T17:33:09.568 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-07T17:33:09.618 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-07T17:33:09.835 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T17:33:09.835 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T17:33:09.836 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: falling back to frontend: Readline 2022-03-07T17:33:09.843 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T17:33:09.843 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T17:33:09.844 INFO:teuthology.orchestra.run.smithi085.stderr:debconf: falling back to frontend: Teletype 2022-03-07T17:33:09.849 INFO:teuthology.orchestra.run.smithi085.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T17:33:09.851 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T17:33:09.852 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T17:33:09.852 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Readline 2022-03-07T17:33:09.861 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T17:33:09.861 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T17:33:09.862 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Teletype 2022-03-07T17:33:09.867 INFO:teuthology.orchestra.run.smithi066.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T17:33:09.900 INFO:teuthology.orchestra.run.smithi085.stdout:Fetched 73.6 MB in 3s (24.7 MB/s) 2022-03-07T17:33:09.918 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 73.6 MB in 3s (24.3 MB/s) 2022-03-07T17:33:09.929 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cri-tools. 2022-03-07T17:33:09.942 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-07T17:33:09.979 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-07T17:33:09.986 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package conntrack. 2022-03-07T17:33:10.011 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package conntrack. 2022-03-07T17:33:10.029 INFO:teuthology.orchestra.run.smithi085.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-07T17:33:10.032 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-07T17:33:10.056 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-07T17:33:10.059 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-07T17:33:10.076 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-07T17:33:10.118 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-07T17:33:10.388 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package cri-tools. 2022-03-07T17:33:10.401 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-07T17:33:10.438 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-07T17:33:10.521 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package cri-tools. 2022-03-07T17:33:10.534 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-07T17:33:10.571 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-07T17:33:11.082 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ebtables. 2022-03-07T17:33:11.094 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-07T17:33:11.148 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-07T17:33:11.409 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-07T17:33:11.422 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-07T17:33:11.459 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-07T17:33:11.499 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package ebtables. 2022-03-07T17:33:11.512 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-07T17:33:11.565 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-07T17:33:11.574 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ebtables. 2022-03-07T17:33:11.589 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-07T17:33:11.641 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-07T17:33:11.843 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-07T17:33:11.858 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-07T17:33:11.902 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-07T17:33:11.902 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-07T17:33:11.915 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-07T17:33:11.960 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-07T17:33:12.928 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package socat. 2022-03-07T17:33:12.941 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-07T17:33:12.970 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-07T17:33:13.354 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package socat. 2022-03-07T17:33:13.369 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-07T17:33:13.381 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubelet. 2022-03-07T17:33:13.394 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-07T17:33:13.412 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-07T17:33:13.429 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package socat. 2022-03-07T17:33:13.431 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-07T17:33:13.443 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-07T17:33:13.487 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-07T17:33:13.783 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package kubelet. 2022-03-07T17:33:13.798 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-07T17:33:13.832 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubelet. 2022-03-07T17:33:13.833 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-07T17:33:13.846 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-07T17:33:13.891 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-07T17:33:15.891 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubectl. 2022-03-07T17:33:15.904 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-07T17:33:15.940 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-07T17:33:16.225 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package kubectl. 2022-03-07T17:33:16.238 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-07T17:33:16.283 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-07T17:33:16.296 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubectl. 2022-03-07T17:33:16.309 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-07T17:33:16.346 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-07T17:33:17.143 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubeadm. 2022-03-07T17:33:17.156 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-07T17:33:17.193 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-07T17:33:17.452 INFO:teuthology.orchestra.run.smithi085.stdout:Selecting previously unselected package kubeadm. 2022-03-07T17:33:17.465 INFO:teuthology.orchestra.run.smithi085.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-07T17:33:17.510 INFO:teuthology.orchestra.run.smithi085.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-07T17:33:17.532 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubeadm. 2022-03-07T17:33:17.547 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-07T17:33:17.582 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-07T17:33:18.350 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-07T17:33:18.474 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-07T17:33:18.600 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-07T17:33:18.657 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-07T17:33:18.731 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-07T17:33:18.780 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-07T17:33:18.854 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-07T17:33:18.909 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-07T17:33:18.980 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-07T17:33:19.594 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-07T17:33:19.714 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-07T17:33:19.759 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-07T17:33:19.803 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-07T17:33:19.824 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-07T17:33:19.886 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-07T17:33:19.923 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-07T17:33:19.951 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-07T17:33:20.014 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-07T17:33:20.050 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-07T17:33:20.123 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-07T17:33:20.177 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-07T17:33:20.187 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-07T17:33:20.353 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-07T17:33:20.396 INFO:teuthology.orchestra.run.smithi085.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-07T17:33:20.702 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-07T17:33:20.825 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-07T17:33:20.845 INFO:teuthology.orchestra.run.smithi085.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-07T17:33:21.129 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T17:33:21.231 INFO:teuthology.orchestra.run.smithi085.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T17:33:21.243 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T17:33:22.116 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-07T17:33:22.118 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-07T17:33:22.121 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-07T17:33:35.940 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-07T17:33:36.307 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-07T17:33:36.709 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-07T17:33:38.792 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-07T17:33:39.294 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-07T17:33:39.886 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-07T17:33:41.114 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-07T17:33:41.715 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-07T17:33:42.825 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-07T17:33:45.222 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-07T17:33:45.806 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-07T17:33:46.714 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-07T17:33:46.815 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-07T17:33:47.264 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-07T17:33:48.441 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-07T17:33:52.795 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-07T17:33:54.278 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-07T17:33:54.781 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-07T17:33:54.887 INFO:teuthology.orchestra.run.smithi085.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-07T17:33:56.230 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-07T17:33:57.189 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-07T17:33:57.193 INFO:tasks.kubeadm:Token: abcdef.buku0paaauptdld8 2022-03-07T17:33:57.194 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): '172.21.15.57', Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): '172.21.15.66', Remote(name='ubuntu@smithi085.front.sepia.ceph.com'): '172.21.15.85'} 2022-03-07T17:33:57.194 DEBUG:teuthology.orchestra.run.smithi057:> sudo kubeadm init --node-name smithi057 --token abcdef.buku0paaauptdld8 --pod-network-cidr 10.249.192.0/21 2022-03-07T17:33:57.525 INFO:teuthology.orchestra.run.smithi057.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-07T17:33:57.526 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] Running pre-flight checks 2022-03-07T17:33:57.979 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-07T17:33:57.979 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-07T17:33:57.979 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-07T17:33:58.277 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-07T17:33:58.545 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "ca" certificate and key 2022-03-07T17:33:58.614 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "apiserver" certificate and key 2022-03-07T17:33:58.614 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi057] and IPs [10.96.0.1 172.21.15.57] 2022-03-07T17:33:58.793 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-07T17:33:58.968 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-07T17:33:59.497 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-07T17:33:59.789 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-07T17:33:59.884 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-07T17:33:59.885 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi057] and IPs [172.21.15.57 127.0.0.1 ::1] 2022-03-07T17:33:59.973 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-07T17:33:59.974 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi057] and IPs [172.21.15.57 127.0.0.1 ::1] 2022-03-07T17:34:00.223 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-07T17:34:00.387 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-07T17:34:00.504 INFO:teuthology.orchestra.run.smithi057.stdout:[certs] Generating "sa" key and public key 2022-03-07T17:34:00.505 INFO:teuthology.orchestra.run.smithi057.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-07T17:34:00.579 INFO:teuthology.orchestra.run.smithi057.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-07T17:34:00.853 INFO:teuthology.orchestra.run.smithi057.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-07T17:34:01.033 INFO:teuthology.orchestra.run.smithi057.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-07T17:34:01.367 INFO:teuthology.orchestra.run.smithi057.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-07T17:34:01.379 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-07T17:34:01.381 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-07T17:34:01.382 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Starting the kubelet 2022-03-07T17:34:01.608 INFO:teuthology.orchestra.run.smithi057.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-07T17:34:01.608 INFO:teuthology.orchestra.run.smithi057.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-07T17:34:01.612 INFO:teuthology.orchestra.run.smithi057.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-07T17:34:01.614 INFO:teuthology.orchestra.run.smithi057.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-07T17:34:01.615 INFO:teuthology.orchestra.run.smithi057.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-07T17:34:01.618 INFO:teuthology.orchestra.run.smithi057.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-07T17:34:18.121 INFO:teuthology.orchestra.run.smithi057.stdout:[apiclient] All control plane components are healthy after 16.503210 seconds 2022-03-07T17:34:18.122 INFO:teuthology.orchestra.run.smithi057.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-07T17:34:18.143 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-07T17:34:18.144 INFO:teuthology.orchestra.run.smithi057.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-07T17:34:18.759 INFO:teuthology.orchestra.run.smithi057.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-07T17:34:18.759 INFO:teuthology.orchestra.run.smithi057.stdout:[mark-control-plane] Marking the node smithi057 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-07T17:34:18.760 INFO:teuthology.orchestra.run.smithi057.stdout:[mark-control-plane] Marking the node smithi057 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-07T17:34:19.271 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] Using token: abcdef.buku0paaauptdld8 2022-03-07T17:34:19.271 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-07T17:34:19.310 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-07T17:34:19.343 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-07T17:34:19.356 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-07T17:34:19.360 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-07T17:34:19.406 INFO:teuthology.orchestra.run.smithi057.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-07T17:34:19.444 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-07T17:34:19.932 INFO:teuthology.orchestra.run.smithi057.stdout:[addons] Applied essential addon: CoreDNS 2022-03-07T17:34:20.103 INFO:teuthology.orchestra.run.smithi057.stdout:[addons] Applied essential addon: kube-proxy 2022-03-07T17:34:20.104 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.105 INFO:teuthology.orchestra.run.smithi057.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-07T17:34:20.105 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.105 INFO:teuthology.orchestra.run.smithi057.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-07T17:34:20.105 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.106 INFO:teuthology.orchestra.run.smithi057.stdout: mkdir -p $HOME/.kube 2022-03-07T17:34:20.106 INFO:teuthology.orchestra.run.smithi057.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-07T17:34:20.106 INFO:teuthology.orchestra.run.smithi057.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-07T17:34:20.106 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.107 INFO:teuthology.orchestra.run.smithi057.stdout:Alternatively, if you are the root user, you can run: 2022-03-07T17:34:20.107 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.107 INFO:teuthology.orchestra.run.smithi057.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-07T17:34:20.107 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.108 INFO:teuthology.orchestra.run.smithi057.stdout:You should now deploy a pod network to the cluster. 2022-03-07T17:34:20.108 INFO:teuthology.orchestra.run.smithi057.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-07T17:34:20.108 INFO:teuthology.orchestra.run.smithi057.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-07T17:34:20.108 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.109 INFO:teuthology.orchestra.run.smithi057.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-07T17:34:20.109 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:34:20.109 INFO:teuthology.orchestra.run.smithi057.stdout:kubeadm join 172.21.15.57:6443 --token abcdef.buku0paaauptdld8 \ 2022-03-07T17:34:20.109 INFO:teuthology.orchestra.run.smithi057.stdout: --discovery-token-ca-cert-hash sha256:44d2d9402b02c542b60270bee4c8909b8a71ef069678fa14991dca807d8c720f 2022-03-07T17:34:20.111 DEBUG:teuthology.orchestra.run.smithi066:> sudo kubeadm join 172.21.15.57:6443 --node-name smithi066 --token abcdef.buku0paaauptdld8 --discovery-token-unsafe-skip-ca-verification 2022-03-07T17:34:20.115 DEBUG:teuthology.orchestra.run.smithi085:> sudo kubeadm join 172.21.15.57:6443 --node-name smithi085 --token abcdef.buku0paaauptdld8 --discovery-token-unsafe-skip-ca-verification 2022-03-07T17:34:20.157 INFO:teuthology.orchestra.run.smithi066.stdout:[preflight] Running pre-flight checks 2022-03-07T17:34:20.157 INFO:teuthology.orchestra.run.smithi085.stdout:[preflight] Running pre-flight checks 2022-03-07T17:34:32.936 INFO:teuthology.orchestra.run.smithi066.stdout:[preflight] Reading configuration from the cluster... 2022-03-07T17:34:32.936 INFO:teuthology.orchestra.run.smithi066.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-07T17:34:32.950 INFO:teuthology.orchestra.run.smithi085.stdout:[preflight] Reading configuration from the cluster... 2022-03-07T17:34:32.951 INFO:teuthology.orchestra.run.smithi085.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-07T17:34:33.314 INFO:teuthology.orchestra.run.smithi085.stderr:W0307 17:34:33.312375 10615 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-07T17:34:33.315 INFO:teuthology.orchestra.run.smithi066.stderr:W0307 17:34:33.308551 10612 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-07T17:34:33.333 INFO:teuthology.orchestra.run.smithi085.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-07T17:34:33.333 INFO:teuthology.orchestra.run.smithi085.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-07T17:34:33.334 INFO:teuthology.orchestra.run.smithi085.stdout:[kubelet-start] Starting the kubelet 2022-03-07T17:34:33.334 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-07T17:34:33.335 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-07T17:34:33.335 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Starting the kubelet 2022-03-07T17:34:33.547 INFO:teuthology.orchestra.run.smithi085.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-07T17:34:33.549 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-07T17:34:39.088 INFO:teuthology.orchestra.run.smithi085.stdout: 2022-03-07T17:34:39.088 INFO:teuthology.orchestra.run.smithi085.stdout:This node has joined the cluster: 2022-03-07T17:34:39.088 INFO:teuthology.orchestra.run.smithi085.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-07T17:34:39.088 INFO:teuthology.orchestra.run.smithi085.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-07T17:34:39.089 INFO:teuthology.orchestra.run.smithi085.stdout: 2022-03-07T17:34:39.089 INFO:teuthology.orchestra.run.smithi085.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-07T17:34:39.089 INFO:teuthology.orchestra.run.smithi085.stdout: 2022-03-07T17:34:39.094 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-03-07T17:34:39.095 INFO:teuthology.orchestra.run.smithi066.stdout:This node has joined the cluster: 2022-03-07T17:34:39.095 INFO:teuthology.orchestra.run.smithi066.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-07T17:34:39.095 INFO:teuthology.orchestra.run.smithi066.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-07T17:34:39.095 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-03-07T17:34:39.096 INFO:teuthology.orchestra.run.smithi066.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-07T17:34:39.096 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-03-07T17:34:39.102 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:39.102 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-07T17:34:39.117 INFO:tasks.kubeadm:Setting up kubectl 2022-03-07T17:34:39.118 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-07T17:34:39.177 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-07T17:34:39.196 DEBUG:teuthology.orchestra.run.smithi085:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-07T17:34:39.217 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:39.217 DEBUG:teuthology.orchestra.run.smithi057:> dd of=.kube/config 2022-03-07T17:34:39.225 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:39.226 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/root/.kube/config 2022-03-07T17:34:39.239 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:34:39.239 DEBUG:teuthology.orchestra.run.smithi066:> dd of=.kube/config 2022-03-07T17:34:39.249 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:34:39.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/root/.kube/config 2022-03-07T17:34:39.307 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:34:39.308 DEBUG:teuthology.orchestra.run.smithi085:> dd of=.kube/config 2022-03-07T17:34:39.316 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:34:39.317 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/root/.kube/config 2022-03-07T17:34:39.375 DEBUG:teuthology.orchestra.run.smithi057:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-07T17:34:42.096 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-07T17:34:42.163 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-07T17:34:42.209 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-07T17:34:42.311 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-07T17:34:42.351 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-07T17:34:42.407 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-07T17:34:42.525 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-07T17:34:42.559 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-07T17:34:42.601 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-07T17:34:42.680 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-07T17:34:42.752 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-07T17:34:42.843 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-07T17:34:42.862 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-07T17:34:42.894 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-07T17:34:42.931 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-07T17:34:43.027 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-07T17:34:43.061 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-07T17:34:43.179 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-07T17:34:43.211 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-07T17:34:43.273 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-07T17:34:43.360 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-07T17:34:43.436 INFO:teuthology.orchestra.run.smithi057.stdout:namespace/tigera-operator created 2022-03-07T17:34:43.442 INFO:teuthology.orchestra.run.smithi057.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-07T17:34:43.546 INFO:teuthology.orchestra.run.smithi057.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-07T17:34:43.554 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/tigera-operator created 2022-03-07T17:34:43.593 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-07T17:34:43.629 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-07T17:34:43.699 INFO:teuthology.orchestra.run.smithi057.stdout:deployment.apps/tigera-operator created 2022-03-07T17:34:43.702 DEBUG:teuthology.orchestra.run.smithi057:> kubectl create -f - 2022-03-07T17:34:46.607 INFO:teuthology.orchestra.run.smithi057.stdout:installation.operator.tigera.io/default created 2022-03-07T17:34:46.612 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-07T17:34:46.612 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:46.612 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:46.620 INFO:tasks.kubeadm:Scratch devices on smithi057: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T17:34:46.621 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-07T17:34:46.647 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-07T17:34:46.647 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-07T17:34:46.648 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145093 s, 723 MB/s 2022-03-07T17:34:46.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-07T17:34:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-07T17:34:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-07T17:34:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134429 s, 780 MB/s 2022-03-07T17:34:46.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-07T17:34:46.754 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-07T17:34:46.754 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-07T17:34:46.754 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140349 s, 747 MB/s 2022-03-07T17:34:46.756 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-07T17:34:46.782 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-07T17:34:46.782 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-07T17:34:46.783 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130494 s, 804 MB/s 2022-03-07T17:34:46.784 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:34:46.784 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:46.792 INFO:tasks.kubeadm:Scratch devices on smithi066: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T17:34:46.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-07T17:34:46.869 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-07T17:34:46.869 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-07T17:34:46.869 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156541 s, 670 MB/s 2022-03-07T17:34:46.870 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-07T17:34:46.895 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-07T17:34:46.896 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-07T17:34:46.896 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128608 s, 815 MB/s 2022-03-07T17:34:46.897 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-07T17:34:46.921 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-07T17:34:46.922 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-07T17:34:46.922 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125501 s, 836 MB/s 2022-03-07T17:34:46.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-07T17:34:46.948 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-07T17:34:46.948 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-07T17:34:46.949 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125375 s, 836 MB/s 2022-03-07T17:34:46.950 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:34:46.950 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:46.958 INFO:tasks.kubeadm:Scratch devices on smithi085: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T17:34:46.959 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-07T17:34:46.991 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records in 2022-03-07T17:34:46.991 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2022-03-07T17:34:46.992 INFO:teuthology.orchestra.run.smithi085.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0171467 s, 612 MB/s 2022-03-07T17:34:46.993 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-07T17:34:47.072 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records in 2022-03-07T17:34:47.072 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2022-03-07T17:34:47.083 INFO:teuthology.orchestra.run.smithi085.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175105 s, 599 MB/s 2022-03-07T17:34:47.084 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-07T17:34:47.114 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records in 2022-03-07T17:34:47.114 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2022-03-07T17:34:47.115 INFO:teuthology.orchestra.run.smithi085.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0183103 s, 573 MB/s 2022-03-07T17:34:47.116 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-07T17:34:47.142 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records in 2022-03-07T17:34:47.143 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2022-03-07T17:34:47.143 INFO:teuthology.orchestra.run.smithi085.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.017132 s, 612 MB/s 2022-03-07T17:34:47.162 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-07T17:34:47.162 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi085-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi085 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi085-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi085 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi085-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi085 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi085-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi085 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-07T17:34:47.163 DEBUG:teuthology.orchestra.run.smithi057:> kubectl create -f - 2022-03-07T17:34:48.036 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi057-lv-1 created 2022-03-07T17:34:48.041 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi057-lv-2 created 2022-03-07T17:34:48.050 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi057-lv-3 created 2022-03-07T17:34:48.058 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi057-lv-4 created 2022-03-07T17:34:48.123 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi066-lv-1 created 2022-03-07T17:34:48.133 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi066-lv-2 created 2022-03-07T17:34:48.156 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi066-lv-3 created 2022-03-07T17:34:48.167 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi066-lv-4 created 2022-03-07T17:34:48.175 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi085-lv-1 created 2022-03-07T17:34:48.199 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi085-lv-2 created 2022-03-07T17:34:48.211 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi085-lv-3 created 2022-03-07T17:34:48.269 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi085-lv-4 created 2022-03-07T17:34:48.273 INFO:teuthology.orchestra.run.smithi057.stdout:storageclass.storage.k8s.io/scratch created 2022-03-07T17:34:48.277 DEBUG:teuthology.orchestra.run.smithi057:> kubectl taint node smithi057 node-role.kubernetes.io/master- || true 2022-03-07T17:34:48.342 INFO:teuthology.orchestra.run.smithi057.stdout:node/smithi057 untainted 2022-03-07T17:34:48.344 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-07T17:34:48.344 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-07T17:34:48.357 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-07T17:34:48.358 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:48.359 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:48.366 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T17:34:48.367 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-07T17:34:48.416 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-07T17:34:48.416 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:48.416 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-07T17:34:48.417 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:48.417 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-07 17:34:46.651067305 +0000 2022-03-07T17:34:48.417 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-07 17:34:46.651067305 +0000 2022-03-07T17:34:48.417 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-07 17:34:46.651067305 +0000 2022-03-07T17:34:48.418 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-07T17:34:48.425 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-07T17:34:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-07T17:34:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000467453 s, 1.1 MB/s 2022-03-07T17:34:48.430 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-07T17:34:48.437 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-07T17:34:48.438 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:48.438 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-07T17:34:48.438 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:48.438 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-07 17:34:46.679067304 +0000 2022-03-07T17:34:48.439 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-07 17:34:46.679067304 +0000 2022-03-07T17:34:48.439 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-07 17:34:46.679067304 +0000 2022-03-07T17:34:48.439 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-07T17:34:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-07T17:34:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-07T17:34:48.447 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000441228 s, 1.2 MB/s 2022-03-07T17:34:48.450 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-07T17:34:48.456 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-07T17:34:48.457 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:48.457 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-07T17:34:48.457 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:48.457 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-07 17:34:46.755067302 +0000 2022-03-07T17:34:48.458 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-07 17:34:46.755067302 +0000 2022-03-07T17:34:48.458 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-07 17:34:46.755067302 +0000 2022-03-07T17:34:48.458 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-07T17:34:48.467 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-07T17:34:48.467 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-07T17:34:48.467 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000436098 s, 1.2 MB/s 2022-03-07T17:34:48.472 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-07T17:34:48.520 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-07T17:34:48.521 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:48.521 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-07T17:34:48.521 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:48.521 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-07 17:34:46.787067301 +0000 2022-03-07T17:34:48.521 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-07 17:34:46.787067301 +0000 2022-03-07T17:34:48.522 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-07 17:34:46.787067301 +0000 2022-03-07T17:34:48.522 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-07T17:34:48.531 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-07T17:34:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-07T17:34:48.532 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000477584 s, 1.1 MB/s 2022-03-07T17:34:48.537 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-07T17:34:48.642 INFO:teuthology.orchestra.run.smithi057.stdout:loop 2022-03-07T17:34:48.644 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_1... 2022-03-07T17:34:48.644 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-07T17:34:48.668 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-07T17:34:48.697 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_11 2022-03-07T17:34:48.720 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_2... 2022-03-07T17:34:48.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-07T17:34:48.782 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-07T17:34:48.808 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_21 2022-03-07T17:34:48.833 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_3... 2022-03-07T17:34:48.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-07T17:34:48.855 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-07T17:34:48.883 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_31 2022-03-07T17:34:48.906 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_4... 2022-03-07T17:34:48.907 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-07T17:34:48.928 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-07T17:34:48.956 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_41 2022-03-07T17:34:48.981 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:48.981 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:48.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo nvme list 2022-03-07T17:34:49.048 INFO:teuthology.orchestra.run.smithi057.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T17:34:49.049 INFO:teuthology.orchestra.run.smithi057.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T17:34:49.049 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme0n1 BTLE7361049H4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2022-03-07T17:34:49.049 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme1n1 b4bb1e8dedac1723 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.049 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme2n1 073a761c3a34b159 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.050 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme3n1 62803024403cbdfd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.050 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme4n1 319201dd206ca68a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.051 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-07T17:34:49.051 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:49.051 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/scratch_devs 2022-03-07T17:34:49.067 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:34:49.068 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:49.075 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T17:34:49.076 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-07T17:34:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-07T17:34:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:49.083 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-07T17:34:49.083 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:49.084 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-07 17:34:46.864666890 +0000 2022-03-07T17:34:49.084 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-07 17:34:46.864666890 +0000 2022-03-07T17:34:49.084 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-07 17:34:46.864666890 +0000 2022-03-07T17:34:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-07T17:34:49.091 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-07T17:34:49.092 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-07T17:34:49.092 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000595649 s, 860 kB/s 2022-03-07T17:34:49.097 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-07T17:34:49.103 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-07T17:34:49.103 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:49.103 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-07T17:34:49.104 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:49.104 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-07 17:34:46.892666890 +0000 2022-03-07T17:34:49.104 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-07 17:34:46.892666890 +0000 2022-03-07T17:34:49.104 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-07 17:34:46.892666890 +0000 2022-03-07T17:34:49.105 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-07T17:34:49.110 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-07T17:34:49.111 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-07T17:34:49.111 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000472406 s, 1.1 MB/s 2022-03-07T17:34:49.115 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-07T17:34:49.163 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-07T17:34:49.163 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:49.163 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-07T17:34:49.164 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:49.164 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-07 17:34:46.916666889 +0000 2022-03-07T17:34:49.164 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-07 17:34:46.916666889 +0000 2022-03-07T17:34:49.164 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-07 17:34:46.916666889 +0000 2022-03-07T17:34:49.165 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-07T17:34:49.172 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-07T17:34:49.172 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-07T17:34:49.172 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000573521 s, 893 kB/s 2022-03-07T17:34:49.177 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-07T17:34:49.184 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-07T17:34:49.184 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:49.184 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-07T17:34:49.185 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:49.185 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-07 17:34:46.944666888 +0000 2022-03-07T17:34:49.185 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-07 17:34:46.944666888 +0000 2022-03-07T17:34:49.185 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-07 17:34:46.944666888 +0000 2022-03-07T17:34:49.185 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-07T17:34:49.193 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-07T17:34:49.193 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-07T17:34:49.194 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000501985 s, 1.0 MB/s 2022-03-07T17:34:49.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-07T17:34:49.272 INFO:teuthology.orchestra.run.smithi066.stdout:loop 2022-03-07T17:34:49.274 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_1... 2022-03-07T17:34:49.275 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-07T17:34:49.299 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-07T17:34:49.324 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_11 2022-03-07T17:34:49.345 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_2... 2022-03-07T17:34:49.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-07T17:34:49.368 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-07T17:34:49.392 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_21 2022-03-07T17:34:49.415 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_3... 2022-03-07T17:34:49.417 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-07T17:34:49.438 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-07T17:34:49.465 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_31 2022-03-07T17:34:49.487 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_4... 2022-03-07T17:34:49.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-07T17:34:49.553 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-07T17:34:49.577 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_41 2022-03-07T17:34:49.601 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:34:49.601 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:49.607 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme list 2022-03-07T17:34:49.618 INFO:teuthology.orchestra.run.smithi066.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T17:34:49.619 INFO:teuthology.orchestra.run.smithi066.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T17:34:49.619 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme0n1 CVFT53430091400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-07T17:34:49.619 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme1n1 1a43a6d7d7772eba Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.619 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme2n1 54c11125031326f4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.620 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme3n1 f5f72b858ff945e6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:49.621 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-07T17:34:50.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme list 2022-03-07T17:34:50.637 INFO:teuthology.orchestra.run.smithi066.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T17:34:50.638 INFO:teuthology.orchestra.run.smithi066.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T17:34:50.638 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme0n1 CVFT53430091400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-07T17:34:50.638 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme1n1 1a43a6d7d7772eba Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:50.638 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme2n1 54c11125031326f4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:50.639 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme3n1 f5f72b858ff945e6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:50.639 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme4n1 b060d318fe24fe6a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:50.640 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-07T17:34:50.640 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:34:50.640 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/scratch_devs 2022-03-07T17:34:50.655 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:34:50.655 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:50.661 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T17:34:50.662 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-07T17:34:50.710 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-07T17:34:50.710 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:50.710 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 5h/5d Inode: 496 Links: 1 2022-03-07T17:34:50.710 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:50.711 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2022-03-07 17:34:46.988201892 +0000 2022-03-07T17:34:50.711 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2022-03-07 17:34:46.988201892 +0000 2022-03-07T17:34:50.711 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2022-03-07 17:34:46.988201892 +0000 2022-03-07T17:34:50.711 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2022-03-07T17:34:50.717 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2022-03-07T17:34:50.717 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2022-03-07T17:34:50.717 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000576055 s, 889 kB/s 2022-03-07T17:34:50.722 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-07T17:34:50.728 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-07T17:34:50.728 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:50.728 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 5h/5d Inode: 503 Links: 1 2022-03-07T17:34:50.729 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:50.729 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2022-03-07 17:34:47.072201891 +0000 2022-03-07T17:34:50.729 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2022-03-07 17:34:47.072201891 +0000 2022-03-07T17:34:50.729 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2022-03-07 17:34:47.072201891 +0000 2022-03-07T17:34:50.730 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2022-03-07T17:34:50.738 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2022-03-07T17:34:50.738 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2022-03-07T17:34:50.738 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000583687 s, 877 kB/s 2022-03-07T17:34:50.742 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-07T17:34:50.749 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-07T17:34:50.749 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:50.749 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-07T17:34:50.750 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:50.750 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2022-03-07 17:34:47.116201890 +0000 2022-03-07T17:34:50.750 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2022-03-07 17:34:47.116201890 +0000 2022-03-07T17:34:50.750 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2022-03-07 17:34:47.116201890 +0000 2022-03-07T17:34:50.751 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2022-03-07T17:34:50.757 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2022-03-07T17:34:50.757 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2022-03-07T17:34:50.758 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000472772 s, 1.1 MB/s 2022-03-07T17:34:50.762 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-07T17:34:50.768 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-07T17:34:50.769 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T17:34:50.769 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 5h/5d Inode: 515 Links: 1 2022-03-07T17:34:50.769 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T17:34:50.769 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2022-03-07 17:34:47.140201889 +0000 2022-03-07T17:34:50.770 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2022-03-07 17:34:47.140201889 +0000 2022-03-07T17:34:50.770 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2022-03-07 17:34:47.140201889 +0000 2022-03-07T17:34:50.770 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2022-03-07T17:34:50.778 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2022-03-07T17:34:50.778 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2022-03-07T17:34:50.778 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000614866 s, 833 kB/s 2022-03-07T17:34:50.782 DEBUG:teuthology.orchestra.run.smithi085:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-07T17:34:50.869 INFO:teuthology.orchestra.run.smithi085.stdout:loop 2022-03-07T17:34:50.870 INFO:tasks.nvme_loop:Connecting nvme_loop smithi085:/dev/vg_nvme/lv_1... 2022-03-07T17:34:50.871 DEBUG:teuthology.orchestra.run.smithi085:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-07T17:34:50.893 INFO:teuthology.orchestra.run.smithi085.stdout:1 2022-03-07T17:34:50.921 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/vg_nvme/lv_11 2022-03-07T17:34:50.946 INFO:tasks.nvme_loop:Connecting nvme_loop smithi085:/dev/vg_nvme/lv_2... 2022-03-07T17:34:50.947 DEBUG:teuthology.orchestra.run.smithi085:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-07T17:34:50.970 INFO:teuthology.orchestra.run.smithi085.stdout:1 2022-03-07T17:34:50.989 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/vg_nvme/lv_21 2022-03-07T17:34:51.012 INFO:tasks.nvme_loop:Connecting nvme_loop smithi085:/dev/vg_nvme/lv_3... 2022-03-07T17:34:51.013 DEBUG:teuthology.orchestra.run.smithi085:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-07T17:34:51.032 INFO:teuthology.orchestra.run.smithi085.stdout:1 2022-03-07T17:34:51.058 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/vg_nvme/lv_31 2022-03-07T17:34:51.080 INFO:tasks.nvme_loop:Connecting nvme_loop smithi085:/dev/vg_nvme/lv_4... 2022-03-07T17:34:51.081 DEBUG:teuthology.orchestra.run.smithi085:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-07T17:34:51.158 INFO:teuthology.orchestra.run.smithi085.stdout:1 2022-03-07T17:34:51.213 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/vg_nvme/lv_41 2022-03-07T17:34:51.214 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:34:51.214 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:34:51.293 DEBUG:teuthology.orchestra.run.smithi085:> sudo nvme list 2022-03-07T17:34:51.327 INFO:teuthology.orchestra.run.smithi085.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T17:34:51.328 INFO:teuthology.orchestra.run.smithi085.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T17:34:51.328 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/nvme0n1 PHFT620400X7400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-07T17:34:51.328 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/nvme1n1 ff0f588d664cf6b4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:51.329 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/nvme2n1 6446bfaf9801f505 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:51.329 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/nvme3n1 102ea6bdc21f3d09 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:51.329 INFO:teuthology.orchestra.run.smithi085.stdout:/dev/nvme4n1 0fb43b40ac00441a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T17:34:51.331 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-07T17:34:51.332 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:34:51.332 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/scratch_devs 2022-03-07T17:34:51.401 INFO:teuthology.run_tasks:Running task rook... 2022-03-07T17:34:51.535 INFO:tasks.rook:Rook start 2022-03-07T17:34:51.535 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-07T17:34:51.536 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 2022-03-07T17:34:51.536 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-07T17:34:51.536 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-07T17:34:51.567 INFO:teuthology.orchestra.run.smithi057.stderr:Cloning into 'rook'... 2022-03-07T17:34:53.807 INFO:teuthology.orchestra.run.smithi057.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-07T17:34:53.807 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.807 INFO:teuthology.orchestra.run.smithi057.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-07T17:34:53.808 INFO:teuthology.orchestra.run.smithi057.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-07T17:34:53.808 INFO:teuthology.orchestra.run.smithi057.stderr:state without impacting any branches by switching back to a branch. 2022-03-07T17:34:53.808 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.808 INFO:teuthology.orchestra.run.smithi057.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-07T17:34:53.808 INFO:teuthology.orchestra.run.smithi057.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-07T17:34:53.809 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.809 INFO:teuthology.orchestra.run.smithi057.stderr: git switch -c 2022-03-07T17:34:53.809 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.809 INFO:teuthology.orchestra.run.smithi057.stderr:Or undo this operation with: 2022-03-07T17:34:53.810 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.810 INFO:teuthology.orchestra.run.smithi057.stderr: git switch - 2022-03-07T17:34:53.810 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.810 INFO:teuthology.orchestra.run.smithi057.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-07T17:34:53.811 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-07T17:34:53.904 INFO:tasks.rook:/home/teuthworker 2022-03-07T17:34:53.905 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/sbin:/usr/sbin ip addr show 2022-03-07T17:34:53.957 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-07T17:34:53.957 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-07T17:34:53.957 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2022-03-07T17:34:53.958 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2022-03-07T17:34:53.958 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2022-03-07T17:34:53.958 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2022-03-07T17:34:53.958 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-07T17:34:53.959 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2022-03-07T17:34:53.959 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2022-03-07T17:34:53.959 INFO:teuthology.orchestra.run.smithi057.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-07T17:34:53.960 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2022-03-07T17:34:53.960 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2022-03-07T17:34:53.960 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-07T17:34:53.960 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42843sec preferred_lft 42843sec 2022-03-07T17:34:53.961 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link 2022-03-07T17:34:53.961 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2022-03-07T17:34:53.961 INFO:teuthology.orchestra.run.smithi057.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-07T17:34:53.961 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2022-03-07T17:34:53.962 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2022-03-07T17:34:53.962 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-07T17:34:53.962 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2022-03-07T17:34:53.962 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2022-03-07T17:34:53.962 INFO:teuthology.orchestra.run.smithi057.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-07T17:34:53.963 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 02:42:8e:b0:ed:8b brd ff:ff:ff:ff:ff:ff 2022-03-07T17:34:53.963 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-07T17:34:53.963 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2022-03-07T17:34:53.964 DEBUG:teuthology.orchestra.run.smithi057:> which systemctl 2022-03-07T17:34:53.971 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/systemctl 2022-03-07T17:34:53.972 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-07T17:34:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:34:53.976 DEBUG:teuthology.orchestra.run.smithi057:> true 2022-03-07T17:34:54.047 DEBUG:teuthology.orchestra.run.smithi057:> sudo lsb_release -is 2022-03-07T17:34:54.098 INFO:teuthology.orchestra.run.smithi057.stdout:Ubuntu 2022-03-07T17:34:54.103 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-07T17:34:54.104 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-07T17:34:54.126 DEBUG:teuthology.orchestra.remote:smithi057:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-07T17:34:54.141 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:54.142 DEBUG:teuthology.orchestra.run.smithi057:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-07T17:34:54.151 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-07T17:34:54.190 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:34:54.191 DEBUG:teuthology.orchestra.run.smithi057:> dd of=operator.yaml 2022-03-07T17:34:54.200 INFO:tasks.rook:Deploying operator 2022-03-07T17:34:54.219 DEBUG:teuthology.orchestra.remote:smithi057:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-07T17:34:54.254 DEBUG:teuthology.orchestra.remote:smithi057:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-07T17:34:54.267 DEBUG:teuthology.orchestra.run.smithi057:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-07T17:34:54.533 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-07T17:34:54.555 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-07T17:34:54.713 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-07T17:34:54.740 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-07T17:34:54.775 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-07T17:34:54.860 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-07T17:34:54.881 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-07T17:34:54.938 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-07T17:34:55.053 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-07T17:34:55.082 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-07T17:34:55.109 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-07T17:34:55.138 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-07T17:34:55.222 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-07T17:34:55.231 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-07T17:34:55.257 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-07T17:34:55.284 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-07T17:34:55.307 INFO:teuthology.orchestra.run.smithi057.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-07T17:34:55.393 INFO:teuthology.orchestra.run.smithi057.stdout:namespace/rook-ceph created 2022-03-07T17:34:55.433 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-07T17:34:55.458 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-07T17:34:55.483 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-07T17:34:55.552 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-07T17:34:55.557 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-07T17:34:55.583 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T17:34:55.590 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T17:34:55.598 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-07T17:34:55.606 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-07T17:34:55.615 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-07T17:34:55.640 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-ceph-system created 2022-03-07T17:34:55.648 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T17:34:55.703 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T17:34:55.713 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-07T17:34:55.723 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-ceph-osd created 2022-03-07T17:34:55.732 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-ceph-mgr created 2022-03-07T17:34:55.740 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-07T17:34:55.793 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T17:34:55.851 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T17:34:55.859 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-07T17:34:55.865 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-07T17:34:55.873 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-07T17:34:55.882 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-07T17:34:55.908 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T17:34:55.915 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-07T17:34:55.923 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-07T17:34:55.932 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-07T17:34:55.940 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T17:34:55.949 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-07T17:34:56.000 INFO:teuthology.orchestra.run.smithi057.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-07T17:34:56.000 INFO:teuthology.orchestra.run.smithi057.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-07T17:34:56.003 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-07T17:34:56.015 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-07T17:34:56.023 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-07T17:34:56.032 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-07T17:34:56.040 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-07T17:34:56.048 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-07T17:34:56.075 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-07T17:34:56.082 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-07T17:34:56.090 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-07T17:34:56.175 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-07T17:34:56.190 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-07T17:34:56.199 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-07T17:34:56.207 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-07T17:34:56.233 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-07T17:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-07T17:34:56.248 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-07T17:34:56.257 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-07T17:34:56.265 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-07T17:34:56.273 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-07T17:34:56.359 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-07T17:34:56.374 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-07T17:34:56.400 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-07T17:34:56.407 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-07T17:34:56.415 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-07T17:34:56.423 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-07T17:34:56.432 INFO:teuthology.orchestra.run.smithi057.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-07T17:34:56.493 INFO:teuthology.orchestra.run.smithi057.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-07T17:34:56.496 INFO:teuthology.orchestra.run.smithi057.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-07T17:34:56.507 INFO:teuthology.orchestra.run.smithi057.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-07T17:34:56.534 INFO:teuthology.orchestra.run.smithi057.stdout:configmap/rook-ceph-operator-config created 2022-03-07T17:34:56.549 INFO:teuthology.orchestra.run.smithi057.stdout:deployment.apps/rook-ceph-operator created 2022-03-07T17:34:56.556 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:34:56.669 INFO:teuthology.orchestra.run.smithi057.stderr:No resources found in rook-ceph namespace. 2022-03-07T17:35:06.674 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:35:06.745 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:35:06.745 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 Pending 0 10s 2022-03-07T17:35:16.749 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:35:16.960 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:35:16.960 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 Pending 0 20s 2022-03-07T17:35:26.963 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:35:27.022 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:35:27.023 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 Pending 0 31s 2022-03-07T17:35:37.024 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:35:37.088 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:35:37.088 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 ContainerCreating 0 41s 2022-03-07T17:35:47.093 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:35:47.152 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:35:47.152 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 ContainerCreating 0 51s 2022-03-07T17:35:57.155 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:35:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:35:57.234 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 ContainerCreating 0 61s 2022-03-07T17:36:07.238 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:36:07.550 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:07.550 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 ContainerCreating 0 71s 2022-03-07T17:36:17.553 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:36:17.624 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:17.624 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 0/1 ContainerCreating 0 81s 2022-03-07T17:36:27.628 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T17:36:27.716 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:27.716 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-operator-6b496c8959-ndm2b 1/1 Running 0 91s 2022-03-07T17:36:27.720 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-ndm2b 2022-03-07T17:36:27.729 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:36:27.729 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:36:27.779 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:36:27.779 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:36:27.787 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:36:27.787 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T17:36:27.797 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-07T17:36:27.797 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-07T17:36:27.798 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-07T17:36:27.798 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-07T17:36:27.798 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-07T17:36:27.799 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-07T17:36:27.799 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-07T17:36:27.800 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-07T17:36:27.802 DEBUG:teuthology.orchestra.run.smithi057:> kubectl create -f - 2022-03-07T17:36:27.811 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:19.737701 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-07T17:36:27.811 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:19.737790 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-07T17:36:27.811 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:19.737794 I | cephcmd: starting Rook-Ceph operator 2022-03-07T17:36:27.812 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:19.909497 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-07T17:36:27.812 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:19.915078 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-07T17:36:27.812 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:19.964942 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-07T17:36:27.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.069475 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-07T17:36:27.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.071953 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-07T17:36:27.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.075241 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-07T17:36:27.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.075312 I | operator: watching all namespaces for ceph cluster CRs 2022-03-07T17:36:27.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.075460 I | operator: setting up the controller-runtime manager 2022-03-07T17:36:27.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.077750 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-07T17:36:27.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.084482 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-07T17:36:27.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484103 I | ceph-cluster-controller: successfully started 2022-03-07T17:36:27.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484188 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-07T17:36:27.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484210 I | ceph-crashcollector-controller: successfully started 2022-03-07T17:36:27.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484280 I | ceph-block-pool-controller: successfully started 2022-03-07T17:36:27.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484354 I | ceph-object-store-user-controller: successfully started 2022-03-07T17:36:27.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484428 I | ceph-object-realm-controller: successfully started 2022-03-07T17:36:27.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484509 I | ceph-object-zonegroup-controller: successfully started 2022-03-07T17:36:27.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484585 I | ceph-object-zone-controller: successfully started 2022-03-07T17:36:27.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484748 I | ceph-object-controller: successfully started 2022-03-07T17:36:27.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484840 I | ceph-file-controller: successfully started 2022-03-07T17:36:27.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.484922 I | ceph-nfs-controller: successfully started 2022-03-07T17:36:27.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.485005 I | ceph-rbd-mirror-controller: successfully started 2022-03-07T17:36:27.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.485087 I | ceph-client-controller: successfully started 2022-03-07T17:36:27.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.485163 I | ceph-filesystem-mirror-controller: successfully started 2022-03-07T17:36:27.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:20.486613 I | operator: starting the controller-runtime manager 2022-03-07T17:36:28.788 INFO:teuthology.orchestra.run.smithi057.stdout:configmap/rook-config-override created 2022-03-07T17:36:28.792 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-07T17:36:28.792 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:36:28.793 DEBUG:teuthology.orchestra.run.smithi057:> dd of=cluster.yaml 2022-03-07T17:36:28.799 DEBUG:teuthology.orchestra.run.smithi057:> kubectl create -f cluster.yaml 2022-03-07T17:36:31.237 INFO:teuthology.orchestra.run.smithi057.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-07T17:36:31.250 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.242377 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-07T17:36:31.250 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.242474 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-07T17:36:31.257 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.252842 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:36:31.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.253629 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-07T17:36:31.259 INFO:tasks.rook.operator.smithi057.stdout:   ... // 3 identical fields 2022-03-07T17:36:31.260 INFO:tasks.rook.operator.smithi057.stdout:   Labels: nil, 2022-03-07T17:36:31.260 INFO:tasks.rook.operator.smithi057.stdout:   Placement: nil, 2022-03-07T17:36:31.262 INFO:tasks.rook.operator.smithi057.stdout:   Network: v1.NetworkSpec{ 2022-03-07T17:36:31.262 INFO:tasks.rook.operator.smithi057.stdout:   Provider: "", 2022-03-07T17:36:31.262 INFO:tasks.rook.operator.smithi057.stdout:   Selectors: nil, 2022-03-07T17:36:31.263 INFO:tasks.rook.operator.smithi057.stdout:   HostNetwork: false, 2022-03-07T17:36:31.263 INFO:tasks.rook.operator.smithi057.stdout:-  IPFamily: "", 2022-03-07T17:36:31.263 INFO:tasks.rook.operator.smithi057.stdout:+  IPFamily: "IPv4", 2022-03-07T17:36:31.263 INFO:tasks.rook.operator.smithi057.stdout:   DualStack: false, 2022-03-07T17:36:31.263 INFO:tasks.rook.operator.smithi057.stdout:   }, 2022-03-07T17:36:31.264 INFO:tasks.rook.operator.smithi057.stdout:   Resources: nil, 2022-03-07T17:36:31.264 INFO:tasks.rook.operator.smithi057.stdout:   PriorityClassNames: nil, 2022-03-07T17:36:31.264 INFO:tasks.rook.operator.smithi057.stdout:   ... // 16 identical fields 2022-03-07T17:36:31.264 INFO:tasks.rook.operator.smithi057.stdout:  } 2022-03-07T17:36:31.265 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.256090 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-07T17:36:31.266 DEBUG:teuthology.orchestra.remote:smithi057:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-07T17:36:31.277 DEBUG:teuthology.orchestra.run.smithi057:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-07T17:36:31.307 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.302829 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-07T17:36:31.336 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.332227 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-07T17:36:31.336 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.332257 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-07T17:36:31.340 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.334479 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-07T17:36:31.343 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.336660 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-07T17:36:31.344 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.338397 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-07T17:36:31.346 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.340634 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-07T17:36:31.346 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.342687 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-07T17:36:31.352 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.348552 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:36:31.479 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.475382 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-07T17:36:31.532 INFO:teuthology.orchestra.run.smithi057.stdout:deployment.apps/rook-ceph-tools created 2022-03-07T17:36:31.535 INFO:tasks.rook:Waiting for tools container to start 2022-03-07T17:36:31.536 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:36:31.670 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:31.670 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 Pending 0 0s 2022-03-07T17:36:31.957 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:31.952762 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-07T17:36:32.262 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:32.258732 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-07T17:36:32.462 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:32.457949 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-07T17:36:32.661 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:32.657767 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-07T17:36:32.861 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:32.857370 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-07T17:36:33.071 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:33.066241 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-07T17:36:33.262 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:33.257871 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-07T17:36:33.263 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:33.257906 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-07T17:36:33.468 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:36:33.464770 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T17:36:36.672 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:36:36.739 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:36.739 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 5s 2022-03-07T17:36:41.743 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:36:41.815 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:41.816 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 10s 2022-03-07T17:36:46.819 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:36:46.881 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:46.881 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 15s 2022-03-07T17:36:51.883 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:36:51.950 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:51.950 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 20s 2022-03-07T17:36:56.952 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:36:57.025 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:36:57.025 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 26s 2022-03-07T17:37:02.029 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:02.100 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:02.101 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 31s 2022-03-07T17:37:07.104 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:07.173 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:07.173 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 36s 2022-03-07T17:37:12.176 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:12.244 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:12.244 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 41s 2022-03-07T17:37:17.246 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:17.314 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:17.315 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 46s 2022-03-07T17:37:22.317 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:22.410 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:22.411 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 51s 2022-03-07T17:37:27.413 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:27.488 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:27.488 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 56s 2022-03-07T17:37:32.492 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:32.563 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:32.564 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 61s 2022-03-07T17:37:34.669 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.665657 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:37:34.742 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.665691 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:37:34.742 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.665706 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:37:34.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.673731 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:37:34.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.812283 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-07 17:37:34.699696103 +0000 UTC m=+75.009040126 LastTransitionTime:2022-03-07 17:37:34.699696 +0000 UTC m=+75.009040022}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:37:34.854 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.851348 I | op-mon: start running mons 2022-03-07T17:37:34.914 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.911146 I | op-mon: creating mon secrets for a new cluster 2022-03-07T17:37:34.945 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.942181 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-07T17:37:34.949 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:34.946340 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-07T17:37:35.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:35.210928 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:37:35.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:35.211209 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:37:36.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:36.212509 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-07T17:37:36.613 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:36.610698 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-07T17:37:37.052 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:37.049456 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T17:37:37.269 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:37.266426 I | op-mon: targeting the mon count 3 2022-03-07T17:37:37.281 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:37.278273 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-07T17:37:37.285 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:37.278374 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:37:37.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:37.410221 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T17:37:37.566 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:37.663 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:37.663 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 0/1 ContainerCreating 0 66s 2022-03-07T17:37:38.036 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.033511 I | op-mon: parsing mon endpoints: 2022-03-07T17:37:38.037 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.033542 W | op-mon: ignoring invalid monitor 2022-03-07T17:37:38.039 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.033704 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T17:37:38.046 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.043796 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:37:38.256 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.253631 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T17:37:38.539 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.536239 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-07T17:37:38.694 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.692001 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-07T17:37:38.699 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.697001 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T17:37:38.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:38.832481 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-07T17:37:39.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:39.011709 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-07T17:37:39.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:39.211243 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-07T17:37:39.427 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:39.424031 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-07T17:37:39.628 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:39.625295 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-07T17:37:39.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:39.811319 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T17:37:40.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:40.012002 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T17:37:40.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:40.012030 I | ceph-csi: Kubernetes version is 1.23 2022-03-07T17:37:40.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:40.212104 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-07T17:37:40.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:40.411200 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-07T17:37:40.704 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:40.701601 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-07T17:37:40.844 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:40.841950 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T17:37:41.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:41.211472 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T17:37:41.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:41.411111 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-07T17:37:41.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:41.612214 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T17:37:41.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:41.811348 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-07T17:37:42.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:42.012602 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T17:37:42.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:42.211377 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-07T17:37:42.438 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:42.435397 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T17:37:42.665 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T17:37:42.696 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:42.691396 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T17:37:42.735 INFO:teuthology.orchestra.run.smithi057.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T17:37:42.735 INFO:teuthology.orchestra.run.smithi057.stdout:rook-ceph-tools-7677f7f4b9-mchzd 1/1 Running 0 71s 2022-03-07T17:37:42.736 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-07T17:37:42.737 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:37:43.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:43.033687 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-07T17:37:43.057 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:43.052832 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-07T17:37:43.790 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:43.787959 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-07T17:37:43.799 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:43.796479 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T17:37:43.879 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:43.876262 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-07T17:37:44.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:44.023420 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T17:37:44.056 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:44.053179 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T17:37:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:37:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:37:44.140+0000 7f3ba9d04700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:37:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:37:44.140+0000 7f3ba9d04700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:37:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:37:44.150 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:37:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:37:44.316 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:44.312877 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-07T17:37:44.366 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:44.363421 I | ceph-csi: successfully started CSI CephFS driver 2022-03-07T17:37:44.637 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:44.634135 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T17:37:44.644 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:44.642195 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-07T17:37:45.303 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:45.299986 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-07T17:37:45.307 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:45.305207 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-07T17:37:47.944 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:47.940830 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:37:47.944 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:47.940891 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:37:47.945 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:47.940910 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:37:47.999 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:47.996388 I | op-mon: parsing mon endpoints: 2022-03-07T17:37:47.999 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:47.996527 W | op-mon: ignoring invalid monitor 2022-03-07T17:37:48.082 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:48.039637 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:37:48.084 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:48.079905 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:37:48.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:48.194984 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:37:48.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:37:48.188+0000 7f54a58c9700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:37:48.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:37:48.188+0000 7f54a58c9700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:37:48.199 INFO:tasks.rook.operator.smithi057.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-03-07T17:37:48.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:48.195021 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:37:48.925 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:48.921090 I | op-mon: start running mons 2022-03-07T17:37:49.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.146156 I | op-mon: parsing mon endpoints: 2022-03-07T17:37:49.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.146188 W | op-mon: ignoring invalid monitor 2022-03-07T17:37:49.288 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.285335 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-07T17:37:49.296 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.293640 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:37:49.297 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.293934 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:37:49.728 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.725797 I | op-mon: targeting the mon count 3 2022-03-07T17:37:49.734 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.731962 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-07T17:37:49.795 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.792360 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-07T17:37:49.950 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:49.947933 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-07T17:37:50.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:50.774192 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi057 2022-03-07T17:37:50.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:50.774223 I | op-mon: assignmon: mon b assigned to node smithi057 2022-03-07T17:37:50.778 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:50.774236 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi066 2022-03-07T17:37:50.778 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:50.774248 I | op-mon: assignmon: mon c assigned to node smithi066 2022-03-07T17:37:54.154 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:37:54.453 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:37:54.453 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:37:54.444+0000 7fae5ffff700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:37:54.453 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:37:54.444+0000 7fae5ffff700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:37:54.454 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:37:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:37:54.461 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:37:54.966 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:54.964283 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi085 2022-03-07T17:37:54.967 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:54.964309 I | op-mon: assignmon: mon a assigned to node smithi085 2022-03-07T17:37:54.972 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:54.969513 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-07T17:37:54.976 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:54.974331 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-07T17:37:55.024 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.020323 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-07T17:37:55.109 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.106722 I | op-mon: creating mon a 2022-03-07T17:37:55.300 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.295775 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:37:55.474 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.471598 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-07T17:37:55.477 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.472223 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-07T17:37:55.477 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.472586 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789"]}] data:a=10.102.1.250:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:-1] 2022-03-07T17:37:55.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.612307 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:37:55.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.612550 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:37:55.661 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.658616 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-07T17:37:55.699 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:55.696262 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-07T17:37:56.159 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:56.157024 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789"]}] data:a=10.102.1.250:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:0] 2022-03-07T17:37:56.160 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:56.157059 I | op-mon: waiting for mon quorum with [a] 2022-03-07T17:37:56.472 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:56.469805 I | op-mon: mon a is not yet running 2022-03-07T17:37:56.472 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:37:56.469840 I | op-mon: mons running: [] 2022-03-07T17:38:04.462 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:38:04.744 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:38:04.745 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:04.736+0000 7f0a55ac7700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:38:04.745 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:04.736+0000 7f0a55ac7700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:38:04.746 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:38:04.748 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:38:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:38:08.919 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:08.916768 I | op-mon: Monitors in quorum: [a] 2022-03-07T17:38:08.919 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:08.916789 I | op-mon: mons created: 1 2022-03-07T17:38:09.141 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.138670 I | op-mon: waiting for mon quorum with [a] 2022-03-07T17:38:09.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.146892 I | op-mon: mons running: [a] 2022-03-07T17:38:09.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.375314 I | op-mon: Monitors in quorum: [a] 2022-03-07T17:38:09.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.375349 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:38:09.682 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.679580 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:38:09.682 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.679615 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:38:09.998 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.995343 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:38:09.998 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:09.995377 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:38:10.301 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:10.299316 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:38:10.302 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:10.299353 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:38:10.585 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:10.582315 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:38:10.585 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:10.582354 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:38:10.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:10.866722 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:38:10.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:10.866757 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:38:11.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.088400 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:38:11.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.088448 I | op-mon: creating mon b 2022-03-07T17:38:11.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.203379 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:38:11.240 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.237416 I | op-mon: mon "b" endpoint is [v2:10.103.120.199:3300,v1:10.103.120.199:6789] 2022-03-07T17:38:11.372 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.369965 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789"]}] data:a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:0] 2022-03-07T17:38:11.385 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.383090 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:38:11.385 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.383264 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:38:11.730 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.728190 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T17:38:11.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.765709 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T17:38:11.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:11.765741 I | op-mon: waiting for mon quorum with [a b] 2022-03-07T17:38:12.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:12.096672 I | op-mon: mon b is not yet running 2022-03-07T17:38:12.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:12.096706 I | op-mon: mons running: [a] 2022-03-07T17:38:12.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:12.323878 I | op-mon: Monitors in quorum: [a] 2022-03-07T17:38:12.423 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:12.420785 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-07T17:38:13.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.311459 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789"]}] data:a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:1] 2022-03-07T17:38:13.314 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.311508 I | op-mon: waiting for mon quorum with [a b] 2022-03-07T17:38:13.539 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.537298 I | op-mon: mon b is not yet running 2022-03-07T17:38:13.540 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.537341 I | op-mon: mons running: [a] 2022-03-07T17:38:13.770 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.768490 I | op-mon: Monitors in quorum: [a] 2022-03-07T17:38:13.772 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.768515 I | op-mon: mons created: 2 2022-03-07T17:38:13.996 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:13.994351 I | op-mon: waiting for mon quorum with [a b] 2022-03-07T17:38:14.009 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:14.007287 I | op-mon: mon b is not yet running 2022-03-07T17:38:14.010 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:14.007316 I | op-mon: mons running: [a] 2022-03-07T17:38:14.752 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:38:15.069 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:38:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:15.060+0000 7f1f648b4700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:38:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:15.060+0000 7f1f648b4700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:38:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:38:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:38:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:38:19.023 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:19.021656 I | op-mon: mon b is not yet running 2022-03-07T17:38:19.024 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:19.021679 I | op-mon: mons running: [a] 2022-03-07T17:38:24.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:24.036249 I | op-mon: mon b is not yet running 2022-03-07T17:38:24.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:24.036285 I | op-mon: mons running: [a] 2022-03-07T17:38:25.075 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:38:25.390 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:38:25.390 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:25.384+0000 7f4252491700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:38:25.391 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:25.384+0000 7f4252491700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:38:25.391 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:38:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:38:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:38:29.054 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:29.052403 I | op-mon: mon b is not yet running 2022-03-07T17:38:29.055 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:29.052434 I | op-mon: mons running: [a] 2022-03-07T17:38:34.421 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:34.419448 I | op-mon: mon b is not yet running 2022-03-07T17:38:34.422 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:34.419471 I | op-mon: mons running: [a] 2022-03-07T17:38:35.397 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:38:35.716 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:38:35.716 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:35.708+0000 7fe964930700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:38:35.716 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:35.708+0000 7fe964930700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:38:35.717 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:38:35.719 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:38:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:38:39.899 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:39.896951 I | op-mon: mon b is not yet running 2022-03-07T17:38:39.899 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:39.896974 I | op-mon: mons running: [a] 2022-03-07T17:38:45.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:45.212248 I | op-mon: mon b is not yet running 2022-03-07T17:38:45.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:45.212273 I | op-mon: mons running: [a] 2022-03-07T17:38:45.722 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:38:46.303 INFO:teuthology.orchestra.run.smithi057.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-07T17:38:46.304 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:46.296+0000 7f2ac1604700 -1 failed for service _ceph-mon._tcp 2022-03-07T17:38:46.304 INFO:teuthology.orchestra.run.smithi057.stderr:2022-03-07T17:38:46.296+0000 7f2ac1604700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-07T17:38:46.304 INFO:teuthology.orchestra.run.smithi057.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-07T17:38:46.307 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 1 2022-03-07T17:38:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T17:38:50.293 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:50.291444 I | op-mon: mon b is not yet running 2022-03-07T17:38:50.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:50.291472 I | op-mon: mons running: [a] 2022-03-07T17:38:55.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:55.309652 I | op-mon: mons running: [a b] 2022-03-07T17:38:56.312 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:38:56.977 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:56.974867 I | op-mon: Monitors in quorum: [a b] 2022-03-07T17:38:56.977 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:56.974909 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:38:57.103 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:38:57.104 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:38:57.104 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:38:57.104 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:38:57.105 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:38:57.105 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:38:57.105 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:38:57.105 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:38:57.106 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:38:57.106 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:38:57.107 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:38:57.107 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:38:57.107 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:38:57.107 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:38:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:38:57.201 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.199575 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:38:57.202 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.199612 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:38:57.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.412059 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:38:57.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.412096 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:38:57.635 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.633503 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:38:57.635 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.633524 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:38:57.855 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.852585 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:38:57.855 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:57.852625 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:38:58.082 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.081014 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:38:58.083 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.081042 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:38:58.304 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.303002 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:38:58.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.303034 I | op-mon: creating mon c 2022-03-07T17:38:58.455 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.453895 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:38:58.624 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.622775 I | op-mon: mon "b" endpoint is [v2:10.103.120.199:3300,v1:10.103.120.199:6789] 2022-03-07T17:38:58.661 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.659467 I | op-mon: mon "c" endpoint is [v2:10.108.168.68:3300,v1:10.108.168.68:6789] 2022-03-07T17:38:58.713 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:58.712080 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789","10.108.168.68:6789"]}] data:a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:1] 2022-03-07T17:38:59.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:59.309186 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:38:59.314 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:59.309474 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:38:59.761 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:59.759318 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T17:38:59.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:59.767264 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T17:38:59.769 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:38:59.767295 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:39:00.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.311413 I | op-mon: mon c is not yet running 2022-03-07T17:39:00.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.311448 I | op-mon: mons running: [a b] 2022-03-07T17:39:00.541 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.539405 I | op-mon: Monitors in quorum: [a b] 2022-03-07T17:39:00.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.545482 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T17:39:00.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.556041 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T17:39:00.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.556231 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:39:00.912 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.910988 I | op-mon: mon c is not yet running 2022-03-07T17:39:00.913 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:00.911025 I | op-mon: mons running: [a b] 2022-03-07T17:39:01.144 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:01.142161 I | op-mon: Monitors in quorum: [a b] 2022-03-07T17:39:01.229 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:01.227758 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-07T17:39:01.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:01.909296 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789","10.108.168.68:6789"]}] data:b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:39:01.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:01.909336 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:39:02.552 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:02.550924 I | op-mon: mon c is not yet running 2022-03-07T17:39:02.553 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:02.550961 I | op-mon: mons running: [a b] 2022-03-07T17:39:02.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:02.812397 I | op-mon: Monitors in quorum: [a b] 2022-03-07T17:39:02.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:02.812434 I | op-mon: mons created: 3 2022-03-07T17:39:03.034 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:03.033090 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:39:03.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:03.113583 I | op-mon: mon c is not yet running 2022-03-07T17:39:03.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:03.113629 I | op-mon: mons running: [a b] 2022-03-07T17:39:07.110 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:39:07.592 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:39:07.592 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:39:07.593 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:39:07.593 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:39:07.593 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:39:07.593 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:39:07.594 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:39:07.594 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:39:07.594 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:39:07.594 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:39:07.595 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:39:07.595 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:39:07.595 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:39:07.598 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:39:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:39:08.139 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:08.137604 I | op-mon: mon c is not yet running 2022-03-07T17:39:08.139 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:08.137635 I | op-mon: mons running: [a b] 2022-03-07T17:39:13.165 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:13.162634 I | op-mon: mon c is not yet running 2022-03-07T17:39:13.166 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:13.162669 I | op-mon: mons running: [a b] 2022-03-07T17:39:17.602 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:39:18.090 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:39:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:39:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:39:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:39:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:39:18.092 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:39:18.092 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:39:18.092 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:39:18.092 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:39:18.092 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:39:18.093 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:39:18.093 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:39:18.093 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:39:18.095 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:39:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:39:18.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:18.185742 I | op-mon: mon c is not yet running 2022-03-07T17:39:18.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:18.185771 I | op-mon: mons running: [a b] 2022-03-07T17:39:23.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:23.214101 I | op-mon: mon c is not yet running 2022-03-07T17:39:23.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:23.214126 I | op-mon: mons running: [a b] 2022-03-07T17:39:28.099 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:39:28.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:28.234504 I | op-mon: mons running: [a b c] 2022-03-07T17:39:28.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:28.467522 W | op-mon: failed to find initial monitor c in mon map 2022-03-07T17:39:28.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:28.467550 W | op-mon: monitor c is not in quorum list 2022-03-07T17:39:28.575 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:39:28.576 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:39:28.576 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:39:28.576 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:39:28.576 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:39:28.577 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:39:28.577 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:39:28.577 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:39:28.577 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:39:28.577 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:39:28.578 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:39:28.578 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:39:28.578 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:39:28.581 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:39:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:39:33.490 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:33.489024 I | op-mon: mon c is not yet running 2022-03-07T17:39:33.490 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:33.489054 I | op-mon: mons running: [a b] 2022-03-07T17:39:38.511 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:38.509671 I | op-mon: mon c is not yet running 2022-03-07T17:39:38.511 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:38.509699 I | op-mon: mons running: [a b] 2022-03-07T17:39:38.584 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:39:39.064 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:39:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:39:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:39:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:39:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:39:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:39:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:39:39.067 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:39:39.067 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:39:39.067 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:39:39.067 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:39:39.067 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:39:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:39:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:39:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:39:43.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:43.531444 I | op-mon: mon c is not yet running 2022-03-07T17:39:43.533 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:43.531472 I | op-mon: mons running: [a b] 2022-03-07T17:39:48.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:48.553370 I | op-mon: mon c is not yet running 2022-03-07T17:39:48.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:48.553400 I | op-mon: mons running: [a b] 2022-03-07T17:39:49.072 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:39:49.598 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:39:49.599 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:39:49.600 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:39:49.600 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:39:49.600 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:39:49.601 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:39:49.601 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:39:49.602 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:39:49.602 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:39:49.602 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:39:49.603 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:39:49.603 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:39:49.604 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:39:49.605 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:39:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:39:53.574 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:53.573413 I | op-mon: mon c is not yet running 2022-03-07T17:39:53.574 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:53.573450 I | op-mon: mons running: [a b] 2022-03-07T17:39:58.601 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:58.598011 I | op-mon: mon c is not yet running 2022-03-07T17:39:58.601 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:39:58.598045 I | op-mon: mons running: [a b] 2022-03-07T17:39:59.608 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:40:00.096 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:00.096 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:40:00.097 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:40:00.097 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:40:00.097 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:40:00.097 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:40:00.098 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:40:00.098 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:40:00.098 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:40:00.098 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:40:00.098 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:40:00.099 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:40:00.099 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:40:00.101 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:40:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:40:03.617 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:03.616954 I | op-mon: mon c is not yet running 2022-03-07T17:40:03.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:03.616981 I | op-mon: mons running: [a b] 2022-03-07T17:40:08.639 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:08.638362 I | op-mon: mons running: [a b c] 2022-03-07T17:40:10.104 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:40:12.303 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:40:12.305 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:40:12.306 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:40:12.306 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:40:12.306 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:40:12.306 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:40:12.307 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:40:12.307 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:40:12.307 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:40:12.307 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:40:12.307 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:40:12.308 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:40:12.308 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:40:12.308 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:40:12.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:12.482863 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:40:12.484 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:12.482901 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:40:12.716 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:12.715455 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:40:12.716 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:12.715486 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:40:12.961 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:12.959596 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:40:12.962 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:12.959616 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:40:13.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.198753 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:40:13.200 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.198789 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:40:13.454 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.454164 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:40:13.455 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.454185 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:40:13.675 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.674830 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:40:13.676 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.674864 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:40:13.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.899385 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:40:13.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:13.899432 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T17:40:14.289 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:14.288781 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T17:40:14.675 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:14.673492 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T17:40:15.074 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:15.073900 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T17:40:15.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:15.531829 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T17:40:15.533 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:15.531852 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T17:40:15.884 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:15.883774 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T17:40:16.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:16.117387 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T17:40:16.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:16.117424 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T17:40:16.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:16.117435 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T17:40:16.449 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:16.449036 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T17:40:16.464 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:16.463999 I | op-mgr: start running mgr 2022-03-07T17:40:16.465 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:16.464049 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T17:40:22.309 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:40:22.783 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:22.783 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:40:22.784 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:40:22.784 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:40:22.784 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:40:22.784 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:40:22.785 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:40:22.785 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:40:22.785 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:40:22.785 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:40:22.785 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:40:22.786 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:40:22.786 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:40:22.788 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:40:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:40:25.932 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:25.931831 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:40:25.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:25.933849 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-07T17:40:25.960 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:25.959654 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:40:25.982 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:25.981530 I | op-mgr: setting services to point to mgr "a" 2022-03-07T17:40:26.154 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.153277 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:40:26.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.197999 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T17:40:26.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.198025 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-07T17:40:26.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.198034 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T17:40:26.286 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.285914 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T17:40:26.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.289688 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-07T17:40:26.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.289745 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T17:40:26.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.289755 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T17:40:26.427 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.427131 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-07T17:40:26.575 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.574508 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T17:40:26.575 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.574540 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:40:26.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.581796 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-07T17:40:26.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.581827 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-07T17:40:26.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.581852 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-07T17:40:26.693 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.693214 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-07T17:40:26.694 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.693238 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-07T17:40:26.697 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.694858 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-07T17:40:26.697 INFO:tasks.rook.operator.smithi057.stdout:I0307 17:40:26.694993 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-07T17:40:26.697 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.695052 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:40:26.703 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.703418 I | op-mon: parsing mon endpoints: b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 2022-03-07T17:40:26.713 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.713431 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:40:26.811 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:26.811105 I | op-mgr: successful modules: prometheus 2022-03-07T17:40:27.325 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:27.325233 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-07T17:40:27.936 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:27.933513 I | op-mgr: successful modules: dashboard 2022-03-07T17:40:32.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.222296 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:40:32.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.222369 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:40:32.224 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.222385 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:40:32.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.235602 I | op-mon: parsing mon endpoints: b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 2022-03-07T17:40:32.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.323209 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:40:32.324 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.323450 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:40:32.556 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.556058 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:40:32.665 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.665417 I | op-mon: start running mons 2022-03-07T17:40:32.720 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.720666 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-07T17:40:32.769 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.769142 I | op-mon: parsing mon endpoints: b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 2022-03-07T17:40:32.791 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:40:32.841 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.841033 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.120.199:6789","10.108.168.68:6789","10.102.1.250:6789"]}] data:b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:40:32.958 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.958078 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:40:32.958 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:32.958293 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:40:33.111 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:33.110557 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T17:40:33.297 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:33.298 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:40:33.298 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:40:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:40:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:40:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:40:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:40:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:40:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:40:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:40:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:40:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:40:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:40:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:40:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:40:34.158 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.158132 I | op-mon: targeting the mon count 3 2022-03-07T17:40:34.163 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.163020 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:40:34.395 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.394809 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:40:34.395 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.394839 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:40:34.496 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.495181 I | cephclient: command failed for set rook backend. trying again... 2022-03-07T17:40:34.621 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.620928 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:40:34.622 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.620961 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:40:34.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.835126 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:40:34.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:34.835162 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:40:35.060 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.059753 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:40:35.060 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.059792 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:40:35.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.322883 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:40:35.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.322915 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:40:35.551 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.550956 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:40:35.551 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.550984 I | op-mon: checking for basic quorum with existing mons 2022-03-07T17:40:35.655 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.655174 I | op-mon: mon "b" endpoint is [v2:10.103.120.199:3300,v1:10.103.120.199:6789] 2022-03-07T17:40:35.744 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:35.743581 I | op-mon: mon "c" endpoint is [v2:10.108.168.68:3300,v1:10.108.168.68:6789] 2022-03-07T17:40:36.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:36.545398 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:40:37.142 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:37.142412 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.120.199:6789","10.108.168.68:6789","10.102.1.250:6789"]}] data:b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:40:37.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:37.741930 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:40:37.746 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:37.742306 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:40:38.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:38.028605 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-07T17:40:38.203 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:38.202419 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T17:40:38.210 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:38.210644 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T17:40:38.211 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:38.210670 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T17:40:38.747 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:38.747410 I | op-mon: mons running: [b c a] 2022-03-07T17:40:39.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.014994 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:40:39.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.025442 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T17:40:39.034 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.034310 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T17:40:39.034 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.034336 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T17:40:39.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.347131 I | op-mon: mons running: [b c a] 2022-03-07T17:40:39.572 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.572318 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:40:39.577 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.577672 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T17:40:39.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.585912 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T17:40:39.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.585939 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T17:40:39.802 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.802181 I | cephclient: command failed for set rook backend. trying again... 2022-03-07T17:40:39.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:39.948058 I | op-mon: mons running: [b c a] 2022-03-07T17:40:40.172 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:40.172544 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:40:40.173 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:40.172572 I | op-mon: mons created: 3 2022-03-07T17:40:40.438 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:40.437890 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T17:40:40.547 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:40.547479 I | op-mon: mons running: [b c a] 2022-03-07T17:40:40.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:40.768495 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:40:40.769 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:40.768526 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T17:40:41.013 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:41.013693 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T17:40:41.253 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:41.253354 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T17:40:41.510 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:41.509945 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T17:40:42.144 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:42.144492 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T17:40:42.145 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:42.144527 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T17:40:42.612 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:42.611710 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T17:40:42.844 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:42.844122 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T17:40:42.845 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:42.844153 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T17:40:42.845 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:42.844161 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T17:40:43.071 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:43.071666 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T17:40:43.304 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:40:43.306 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:43.306515 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-07T17:40:43.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:43.743337 I | op-mgr: start running mgr 2022-03-07T17:40:43.744 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:43.743380 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T17:40:44.136 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:44.136 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 10 closest matches: 2022-03-07T17:40:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:pg stat 2022-03-07T17:40:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:pg getmap 2022-03-07T17:40:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T17:40:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T17:40:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_pools_json 2022-03-07T17:40:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-pool [...] 2022-03-07T17:40:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-primary [] [...] 2022-03-07T17:40:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls-by-osd [] [...] 2022-03-07T17:40:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:pg ls [] [...] 2022-03-07T17:40:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T17:40:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2022-03-07T17:40:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:command terminated with exit code 22 2022-03-07T17:40:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T17:40:44.401 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.401232 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T17:40:44.410 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.410554 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T17:40:44.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.413335 I | op-mgr: setting services to point to mgr "a" 2022-03-07T17:40:44.720 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.720691 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T17:40:44.721 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.720719 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-07T17:40:44.721 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.720729 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T17:40:44.729 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.728890 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T17:40:44.760 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.760160 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-07T17:40:44.760 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.760189 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T17:40:44.760 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:44.760194 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T17:40:45.174 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:45.174259 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T17:40:45.174 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:45.174286 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:40:45.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:45.187866 I | cephclient: command failed for set rook backend. trying again... 2022-03-07T17:40:45.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:45.508256 I | op-mgr: successful modules: prometheus 2022-03-07T17:40:45.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:45.615291 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T17:40:45.616 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:45.615499 I | op-mgr: successful modules: dashboard 2022-03-07T17:40:46.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:46.824419 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-07T17:40:47.466 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:47.466423 I | op-mgr: successful modules: orchestrator modules 2022-03-07T17:40:48.829 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:48.828673 I | op-mgr: successful modules: balancer 2022-03-07T17:40:50.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:50.686363 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-07T17:40:50.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:50.686393 I | op-mgr: successful modules: orchestrator modules 2022-03-07T17:40:52.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:52.179925 I | op-mgr: successful modules: balancer 2022-03-07T17:40:54.144 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status -f json 2022-03-07T17:40:54.794 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:54.794 INFO:teuthology.orchestra.run.smithi057.stdout:{"available": true, "backend": "rook"} 2022-03-07T17:40:54.818 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-07T17:40:54.819 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-07T17:40:55.478 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch apply osd --all-available-devices 2022-03-07T17:40:56.217 INFO:teuthology.orchestra.run.smithi057.stdout:['Success'] 2022-03-07T17:40:56.221 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:56.220091 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-07T17:40:56.221 INFO:tasks.rook.operator.smithi057.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", AllowUnsupported: true}, 2022-03-07T17:40:56.221 INFO:tasks.rook.operator.smithi057.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-07T17:40:56.221 INFO:tasks.rook.operator.smithi057.stdout:   ... // 3 identical fields 2022-03-07T17:40:56.222 INFO:tasks.rook.operator.smithi057.stdout:   Config: nil, 2022-03-07T17:40:56.222 INFO:tasks.rook.operator.smithi057.stdout:   Selection: {}, 2022-03-07T17:40:56.222 INFO:tasks.rook.operator.smithi057.stdout:-  StorageClassDeviceSets: nil, 2022-03-07T17:40:56.223 INFO:tasks.rook.operator.smithi057.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-07T17:40:56.223 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.224 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi057-lv-1", 2022-03-07T17:40:56.229 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.230 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.230 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.230 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.231 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi057-lv-2", 2022-03-07T17:40:56.231 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.231 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.231 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.231 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.232 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi057-lv-3", 2022-03-07T17:40:56.232 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.232 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.232 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.233 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.233 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi057-lv-4", 2022-03-07T17:40:56.233 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.233 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.234 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.234 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.234 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi066-lv-1", 2022-03-07T17:40:56.234 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.235 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.235 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.235 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.236 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi066-lv-2", 2022-03-07T17:40:56.236 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.237 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.237 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.237 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.238 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi066-lv-3", 2022-03-07T17:40:56.238 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.239 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.239 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.240 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.240 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi066-lv-4", 2022-03-07T17:40:56.240 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.241 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.241 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.241 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.241 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi085-lv-1", 2022-03-07T17:40:56.242 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.242 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.242 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.242 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.243 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi085-lv-2", 2022-03-07T17:40:56.243 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.243 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.243 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.244 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.244 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi085-lv-3", 2022-03-07T17:40:56.244 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.244 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.245 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.245 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:40:56.245 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi085-lv-4", 2022-03-07T17:40:56.245 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:40:56.245 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T17:40:56.246 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.246 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:40:56.246 INFO:tasks.rook.operator.smithi057.stdout:   }, 2022-03-07T17:40:56.246 INFO:tasks.rook.operator.smithi057.stdout:   Annotations: nil, 2022-03-07T17:40:56.247 INFO:tasks.rook.operator.smithi057.stdout:   Labels: nil, 2022-03-07T17:40:56.247 INFO:tasks.rook.operator.smithi057.stdout:   ... // 20 identical fields 2022-03-07T17:40:56.247 INFO:tasks.rook.operator.smithi057.stdout:  } 2022-03-07T17:40:56.247 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:56.220244 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:40:56.249 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-07T17:40:56.249 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:40:56.250 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:56.246167 I | op-mon: parsing mon endpoints: b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 2022-03-07T17:40:56.288 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:40:56.288116 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:40:56.784 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:40:56.807 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T17:40:56.808 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:41:01.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.482874 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:41:01.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.482900 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:41:01.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.482913 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:41:01.489 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.487413 I | op-mon: parsing mon endpoints: b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 2022-03-07T17:41:01.523 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.522910 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:41:01.527 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.523205 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:41:01.953 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:01.953156 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:41:02.049 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:02.049149 I | op-mon: start running mons 2022-03-07T17:41:02.111 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:02.111080 I | op-mon: parsing mon endpoints: b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 2022-03-07T17:41:02.157 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:02.157366 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.108.168.68:6789","10.102.1.250:6789","10.103.120.199:6789"]}] data:a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:41:02.230 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:02.229972 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:41:02.230 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:02.230196 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:41:03.433 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:03.432922 I | op-mon: targeting the mon count 3 2022-03-07T17:41:03.449 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:03.449952 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-07T17:41:04.451 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:04.450926 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:41:04.458 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:04.458333 I | op-mon: parsing mon endpoints: a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 2022-03-07T17:41:04.474 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:04.475022 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:41:04.523 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:04.523831 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-07T17:41:04.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:04.554574 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T17:41:06.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:06.558411 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-07T17:41:06.811 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:41:07.309 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:41:07.332 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T17:41:07.333 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:41:11.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:11.895036 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:41:11.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:11.895051 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:41:11.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:11.895057 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:41:11.916 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:11.916796 I | op-mon: parsing mon endpoints: a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 2022-03-07T17:41:11.994 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:11.995021 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:41:12.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:11.995280 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:41:12.499 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:12.499221 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:41:12.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:12.587569 I | op-mon: start running mons 2022-03-07T17:41:12.610 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:12.610631 I | op-mon: parsing mon endpoints: a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 2022-03-07T17:41:12.626 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:12.627055 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789","10.108.168.68:6789"]}] data:b=10.103.120.199:6789,c=10.108.168.68:6789,a=10.102.1.250:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:41:12.665 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:12.666488 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:41:12.666 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:12.666617 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:41:13.794 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:13.794118 I | op-mon: targeting the mon count 3 2022-03-07T17:41:13.798 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:13.798638 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:41:14.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:14.147411 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:41:14.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:14.147452 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:41:14.506 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:14.506922 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:41:14.507 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:14.506963 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:41:14.854 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:14.854897 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:41:14.855 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:14.854933 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:41:15.221 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:15.221826 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:41:15.222 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:15.221857 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:41:15.562 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:15.562450 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:41:15.562 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:15.562490 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:41:15.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:15.915017 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:41:15.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:15.915047 I | op-mon: checking for basic quorum with existing mons 2022-03-07T17:41:16.043 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:16.043704 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:41:16.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:16.241185 I | op-mon: mon "b" endpoint is [v2:10.103.120.199:3300,v1:10.103.120.199:6789] 2022-03-07T17:41:16.366 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:16.366562 I | op-mon: mon "c" endpoint is [v2:10.108.168.68:3300,v1:10.108.168.68:6789] 2022-03-07T17:41:16.921 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:16.921730 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789","10.108.168.68:6789"]}] data:a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:41:17.334 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:41:17.520 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:17.520731 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:41:17.524 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:17.521109 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:41:17.882 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:41:17.900 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T17:41:17.901 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:41:17.935 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:17.935788 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T17:41:17.943 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:17.943724 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T17:41:17.943 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:17.943747 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:41:18.524 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:18.524919 I | op-mon: mons running: [a b c] 2022-03-07T17:41:18.959 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:18.959386 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:41:18.963 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:18.964521 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T17:41:18.975 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:18.976289 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T17:41:18.975 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:18.976310 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:41:19.123 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:19.123422 I | op-mon: mons running: [a b c] 2022-03-07T17:41:19.542 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:19.542855 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:41:19.549 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:19.549710 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T17:41:19.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:19.555634 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T17:41:19.555 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:19.555655 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:41:19.722 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:19.722755 I | op-mon: mons running: [a b c] 2022-03-07T17:41:20.143 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:20.143758 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:41:20.143 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:20.143791 I | op-mon: mons created: 3 2022-03-07T17:41:20.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:20.558725 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T17:41:20.576 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:20.576863 I | op-mon: mons running: [a b c] 2022-03-07T17:41:20.985 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:20.985241 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:41:20.985 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:20.985292 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T17:41:21.484 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:21.484760 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T17:41:21.913 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:21.913482 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T17:41:22.353 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:22.354222 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T17:41:22.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:22.826840 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T17:41:22.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:22.826864 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T17:41:23.335 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:23.335529 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T17:41:23.748 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:23.748624 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T17:41:23.748 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:23.748661 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T17:41:23.749 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:23.748671 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T17:41:24.171 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.171678 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T17:41:24.216 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.216246 I | op-mgr: start running mgr 2022-03-07T17:41:24.216 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.216279 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T17:41:24.792 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.792936 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T17:41:24.800 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.801681 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T17:41:24.804 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.804099 I | op-mgr: setting services to point to mgr "a" 2022-03-07T17:41:24.891 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.891482 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T17:41:24.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.891520 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T17:41:24.935 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.936086 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T17:41:24.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.939127 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-1" 2022-03-07T17:41:24.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.987697 I | op-osd: successfully provisioned PVC "smithi057-lv-1-data-0ht4bd" 2022-03-07T17:41:24.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:24.987727 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-2" 2022-03-07T17:41:25.234 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.234588 I | op-osd: successfully provisioned PVC "smithi057-lv-2-data-0gkc97" 2022-03-07T17:41:25.234 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.234621 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-3" 2022-03-07T17:41:25.386 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.386423 I | op-osd: successfully provisioned PVC "smithi057-lv-3-data-07k2rn" 2022-03-07T17:41:25.386 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.386462 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-4" 2022-03-07T17:41:25.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.588458 I | op-osd: successfully provisioned PVC "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:41:25.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.588495 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-1" 2022-03-07T17:41:25.786 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.786610 I | op-osd: successfully provisioned PVC "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:41:25.786 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.786679 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-2" 2022-03-07T17:41:25.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.877042 I | op-mgr: successful modules: dashboard 2022-03-07T17:41:25.879 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.880249 I | op-mgr: successful modules: prometheus 2022-03-07T17:41:25.986 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.987273 I | op-osd: successfully provisioned PVC "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:41:25.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:25.987315 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-3" 2022-03-07T17:41:26.049 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.050160 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T17:41:26.211 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.211838 I | op-osd: successfully provisioned PVC "smithi066-lv-3-data-0pc4td" 2022-03-07T17:41:26.211 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.211870 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-4" 2022-03-07T17:41:26.390 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.390921 I | op-osd: successfully provisioned PVC "smithi066-lv-4-data-052zdw" 2022-03-07T17:41:26.390 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.390949 I | op-osd: creating 1 new PVCs for device set "smithi085-lv-1" 2022-03-07T17:41:26.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.414233 I | op-mgr: successful modules: balancer 2022-03-07T17:41:26.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.588069 I | op-osd: successfully provisioned PVC "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:41:26.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.588105 I | op-osd: creating 1 new PVCs for device set "smithi085-lv-2" 2022-03-07T17:41:26.790 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.790075 I | op-osd: successfully provisioned PVC "smithi085-lv-2-data-08x59z" 2022-03-07T17:41:26.790 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.790110 I | op-osd: creating 1 new PVCs for device set "smithi085-lv-3" 2022-03-07T17:41:26.986 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.986936 I | op-osd: successfully provisioned PVC "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:41:26.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:26.986973 I | op-osd: creating 1 new PVCs for device set "smithi085-lv-4" 2022-03-07T17:41:27.216 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:27.217330 I | op-osd: successfully provisioned PVC "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:41:27.245 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:27.246483 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-0ht4bd" 2022-03-07T17:41:27.331 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:27.331869 I | op-mgr: successful modules: orchestrator modules 2022-03-07T17:41:27.623 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:27.623525 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-1-data-0ht4bd" 2022-03-07T17:41:27.623 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:27.623567 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0gkc97" 2022-03-07T17:41:27.903 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:41:28.488 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:28.488330 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-2-data-0gkc97" 2022-03-07T17:41:28.488 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:28.488364 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-07k2rn" 2022-03-07T17:41:28.964 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:41:28.980 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T17:41:28.981 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:41:29.128 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:29.128591 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-3-data-07k2rn" 2022-03-07T17:41:29.128 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:29.128620 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:41:29.569 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:29.569057 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:41:29.569 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:29.569106 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:41:30.536 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:30.536627 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:41:30.537 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:30.536654 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:41:31.239 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:31.239978 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:41:31.240 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:31.240008 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0pc4td" 2022-03-07T17:41:32.006 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:32.006543 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-3-data-0pc4td" 2022-03-07T17:41:32.006 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:32.006576 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-052zdw" 2022-03-07T17:41:32.840 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:32.840168 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-4-data-052zdw" 2022-03-07T17:41:32.840 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:32.840270 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:41:33.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:33.348167 I | op-osd: started OSD provisioning job for PVC "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:41:33.348 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:33.348206 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-2-data-08x59z" 2022-03-07T17:41:33.520 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:33.519102 I | op-osd: started OSD provisioning job for PVC "smithi085-lv-2-data-08x59z" 2022-03-07T17:41:33.521 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:33.519129 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:41:33.958 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:33.957194 I | op-osd: started OSD provisioning job for PVC "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:41:33.959 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:33.957286 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:41:34.930 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:34.930713 I | op-osd: started OSD provisioning job for PVC "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:41:34.930 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:34.930766 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T17:41:34.931 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:34.930777 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T17:41:35.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026382 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-0ht4bd is "starting" 2022-03-07T17:41:35.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026479 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-0gkc97 is "starting" 2022-03-07T17:41:35.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026519 I | op-osd: OSD orchestration status for PVC smithi057-lv-3-data-07k2rn is "starting" 2022-03-07T17:41:35.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026544 I | op-osd: OSD orchestration status for PVC smithi057-lv-4-data-0h9pmf is "starting" 2022-03-07T17:41:35.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026554 I | op-osd: OSD orchestration status for PVC smithi066-lv-1-data-0gmmxm is "starting" 2022-03-07T17:41:35.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026561 I | op-osd: OSD orchestration status for PVC smithi066-lv-2-data-0tbn4m is "starting" 2022-03-07T17:41:35.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026571 I | op-osd: OSD orchestration status for PVC smithi066-lv-3-data-0pc4td is "starting" 2022-03-07T17:41:35.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026578 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-052zdw is "starting" 2022-03-07T17:41:35.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026586 I | op-osd: OSD orchestration status for PVC smithi085-lv-1-data-0vtx9m is "starting" 2022-03-07T17:41:35.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026594 I | op-osd: OSD orchestration status for PVC smithi085-lv-2-data-08x59z is "starting" 2022-03-07T17:41:35.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026602 I | op-osd: OSD orchestration status for PVC smithi085-lv-3-data-0mzbnp is "starting" 2022-03-07T17:41:35.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:35.026610 I | op-osd: OSD orchestration status for PVC smithi085-lv-4-data-0dkcqt is "starting" 2022-03-07T17:41:38.982 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:41:40.853 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:41:40.869 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T17:41:40.870 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:41:47.590 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:47.584123 I | op-osd: OSD orchestration status for node smithi057-lv-1-data-0ht4bd is "orchestrating" 2022-03-07T17:41:48.285 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:48.284489 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-0ht4bd is "orchestrating" 2022-03-07T17:41:48.734 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:48.732804 I | op-osd: OSD orchestration status for node smithi057-lv-2-data-0gkc97 is "orchestrating" 2022-03-07T17:41:49.502 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:49.502616 I | op-osd: OSD orchestration status for node smithi057-lv-3-data-07k2rn is "orchestrating" 2022-03-07T17:41:49.737 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:49.738038 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-0gkc97 is "orchestrating" 2022-03-07T17:41:49.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:49.874287 I | op-osd: OSD orchestration status for PVC smithi057-lv-3-data-07k2rn is "orchestrating" 2022-03-07T17:41:50.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:50.339683 I | op-osd: OSD orchestration status for node smithi066-lv-3-data-0pc4td is "orchestrating" 2022-03-07T17:41:50.852 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:50.851518 I | op-osd: OSD orchestration status for PVC smithi066-lv-3-data-0pc4td is "orchestrating" 2022-03-07T17:41:50.871 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:41:52.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:52.015003 I | op-osd: OSD orchestration status for node smithi057-lv-4-data-0h9pmf is "orchestrating" 2022-03-07T17:41:52.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:52.377934 I | op-osd: OSD orchestration status for PVC smithi057-lv-4-data-0h9pmf is "orchestrating" 2022-03-07T17:41:52.839 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:41:52.868 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"num_osds":3,"num_up_osds":0,"osd_up_since":0,"num_in_osds":3,"osd_in_since":1646674912,"num_remapped_pgs":0} 2022-03-07T17:41:52.869 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:41:52.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:52.910466 I | op-osd: OSD orchestration status for node smithi066-lv-2-data-0tbn4m is "orchestrating" 2022-03-07T17:41:53.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:53.031465 I | op-osd: OSD orchestration status for node smithi085-lv-3-data-0mzbnp is "orchestrating" 2022-03-07T17:41:53.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:53.098068 I | op-osd: OSD orchestration status for node smithi085-lv-1-data-0vtx9m is "orchestrating" 2022-03-07T17:41:53.278 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:53.278050 I | op-osd: OSD orchestration status for PVC smithi066-lv-2-data-0tbn4m is "orchestrating" 2022-03-07T17:41:53.453 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:53.453167 I | op-osd: OSD orchestration status for PVC smithi085-lv-3-data-0mzbnp is "orchestrating" 2022-03-07T17:41:53.641 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:53.641552 I | op-osd: OSD orchestration status for PVC smithi085-lv-1-data-0vtx9m is "orchestrating" 2022-03-07T17:41:54.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:54.482949 I | op-osd: OSD orchestration status for node smithi085-lv-2-data-08x59z is "orchestrating" 2022-03-07T17:41:55.077 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:55.077612 I | op-osd: OSD orchestration status for PVC smithi085-lv-2-data-08x59z is "orchestrating" 2022-03-07T17:41:55.739 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:55.738634 I | op-osd: OSD orchestration status for node smithi066-lv-1-data-0gmmxm is "orchestrating" 2022-03-07T17:41:55.776 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:55.776290 I | op-osd: OSD orchestration status for node smithi066-lv-4-data-052zdw is "orchestrating" 2022-03-07T17:41:56.182 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:56.181459 I | op-osd: OSD orchestration status for PVC smithi066-lv-1-data-0gmmxm is "orchestrating" 2022-03-07T17:41:56.190 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:56.189795 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-052zdw is "orchestrating" 2022-03-07T17:41:56.640 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:56.639962 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-0ht4bd is "completed" 2022-03-07T17:41:56.641 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:56.639996 I | op-osd: creating OSD 0 on PVC "smithi057-lv-1-data-0ht4bd" 2022-03-07T17:41:56.641 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:56.640011 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-0ht4bd" 2022-03-07T17:41:57.725 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:57.722896 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-0gkc97 is "completed" 2022-03-07T17:41:57.728 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:57.723864 I | op-osd: creating OSD 2 on PVC "smithi057-lv-2-data-0gkc97" 2022-03-07T17:41:57.729 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:57.723972 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0gkc97" 2022-03-07T17:41:59.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:59.558697 I | op-osd: OSD orchestration status for PVC smithi057-lv-3-data-07k2rn is "completed" 2022-03-07T17:41:59.561 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:59.558721 I | op-osd: creating OSD 1 on PVC "smithi057-lv-3-data-07k2rn" 2022-03-07T17:41:59.561 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:41:59.558732 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-07k2rn" 2022-03-07T17:42:00.374 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:00.372252 I | op-osd: OSD orchestration status for node smithi085-lv-4-data-0dkcqt is "orchestrating" 2022-03-07T17:42:00.379 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:00.375450 I | op-osd: OSD orchestration status for PVC smithi085-lv-4-data-0dkcqt is "orchestrating" 2022-03-07T17:42:00.379 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:00.375646 I | op-osd: OSD orchestration status for PVC smithi066-lv-3-data-0pc4td is "completed" 2022-03-07T17:42:00.380 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:00.375709 I | op-osd: creating OSD 3 on PVC "smithi066-lv-3-data-0pc4td" 2022-03-07T17:42:00.380 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:00.375746 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0pc4td" 2022-03-07T17:42:02.322 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:02.320070 I | op-osd: OSD orchestration status for PVC smithi057-lv-4-data-0h9pmf is "completed" 2022-03-07T17:42:02.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:02.320105 I | op-osd: creating OSD 4 on PVC "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:42:02.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:02.320118 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:42:02.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:02.815364 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:42:02.870 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:42:03.585 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:03.584233 I | op-osd: OSD orchestration status for PVC smithi066-lv-2-data-0tbn4m is "completed" 2022-03-07T17:42:03.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:03.584268 I | op-osd: creating OSD 5 on PVC "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:42:03.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:03.584280 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:42:04.889 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:42:04.908 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":18,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646674920,"num_remapped_pgs":0} 2022-03-07T17:42:04.909 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:42:05.176 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:05.174230 I | op-osd: OSD orchestration status for PVC smithi085-lv-1-data-0vtx9m is "completed" 2022-03-07T17:42:05.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:05.178995 I | op-osd: creating OSD 7 on PVC "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:42:05.182 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:05.179095 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:42:06.994 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:06.990695 I | op-osd: OSD orchestration status for PVC smithi085-lv-3-data-0mzbnp is "completed" 2022-03-07T17:42:06.995 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:06.990763 I | op-osd: creating OSD 6 on PVC "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:42:06.995 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:06.990782 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:42:08.110 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:08.108145 I | op-osd: OSD orchestration status for PVC smithi085-lv-2-data-08x59z is "completed" 2022-03-07T17:42:08.114 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:08.109684 I | op-osd: creating OSD 8 on PVC "smithi085-lv-2-data-08x59z" 2022-03-07T17:42:08.115 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:08.109888 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-2-data-08x59z" 2022-03-07T17:42:09.700 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:09.693176 I | op-osd: OSD orchestration status for PVC smithi085-lv-4-data-0dkcqt is "completed" 2022-03-07T17:42:09.705 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:09.693767 I | op-osd: creating OSD 11 on PVC "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:42:09.706 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:09.693946 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:42:10.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:10.772733 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-052zdw is "completed" 2022-03-07T17:42:10.782 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:10.773352 I | op-osd: creating OSD 10 on PVC "smithi066-lv-4-data-052zdw" 2022-03-07T17:42:10.782 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:10.773474 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-052zdw" 2022-03-07T17:42:12.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:12.433520 I | op-osd: OSD orchestration status for PVC smithi066-lv-1-data-0gmmxm is "completed" 2022-03-07T17:42:12.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:12.433560 I | op-osd: creating OSD 9 on PVC "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:42:12.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:12.433576 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:42:14.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:14.614300 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T17:42:14.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:42:14.614328 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:42:14.911 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:42:16.363 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:42:16.381 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":18,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646674920,"num_remapped_pgs":0} 2022-03-07T17:42:16.381 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:42:26.385 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:42:28.171 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:42:28.190 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":18,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646674920,"num_remapped_pgs":0} 2022-03-07T17:42:28.191 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T17:42:38.192 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:42:38.731 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:42:38.751 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":22,"num_osds":12,"num_up_osds":1,"osd_up_since":1646674955,"num_in_osds":12,"osd_in_since":1646674920,"num_remapped_pgs":0} 2022-03-07T17:42:38.752 INFO:tasks.rook: have 1/12 OSDs 2022-03-07T17:42:48.753 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:42:49.283 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:42:49.303 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":31,"num_osds":12,"num_up_osds":11,"osd_up_since":1646674968,"num_in_osds":12,"osd_in_since":1646674920,"num_remapped_pgs":0} 2022-03-07T17:42:49.303 INFO:tasks.rook: have 11/12 OSDs 2022-03-07T17:42:59.305 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd stat -f json 2022-03-07T17:42:59.966 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:42:59.990 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"num_osds":12,"num_up_osds":12,"osd_up_since":1646674971,"num_in_osds":12,"osd_in_since":1646674920,"num_remapped_pgs":0} 2022-03-07T17:42:59.991 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-07T17:42:59.991 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- cat /etc/ceph/ceph.conf 2022-03-07T17:43:00.172 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2022-03-07T17:43:00.173 INFO:teuthology.orchestra.run.smithi057.stdout:mon_host = 10.102.1.250:6789,10.103.120.199:6789,10.108.168.68:6789 2022-03-07T17:43:00.173 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:00.173 INFO:teuthology.orchestra.run.smithi057.stdout:[client.admin] 2022-03-07T17:43:00.174 INFO:teuthology.orchestra.run.smithi057.stdout:keyring = /etc/ceph/keyring 2022-03-07T17:43:00.177 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- cat /etc/ceph/keyring 2022-03-07T17:43:00.333 INFO:teuthology.orchestra.run.smithi057.stdout:[client.admin] 2022-03-07T17:43:00.334 INFO:teuthology.orchestra.run.smithi057.stdout:key = AQDeQiZi8Ho7NhAAVvucss1mrcdU6CtNNo6tZw== 2022-03-07T17:43:00.338 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph 2022-03-07T17:43:00.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2022-03-07T17:43:00.367 DEBUG:teuthology.orchestra.run.smithi085:> sudo mkdir -p /etc/ceph 2022-03-07T17:43:00.381 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:43:00.381 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.conf 2022-03-07T17:43:00.398 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:43:00.398 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/keyring 2022-03-07T17:43:00.459 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:43:00.460 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2022-03-07T17:43:00.477 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:43:00.478 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/keyring 2022-03-07T17:43:00.532 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:43:00.532 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/ceph/ceph.conf 2022-03-07T17:43:00.549 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:43:00.550 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/ceph/keyring 2022-03-07T17:43:00.606 INFO:tasks.rook:Setting up client nodes... 2022-03-07T17:43:00.607 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-07T17:43:01.359 INFO:teuthology.orchestra.run.smithi057.stdout:[client.a] 2022-03-07T17:43:01.359 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAlRCZiYNtvDRAABSYyP9mwNc+IH9Y4+kj1cQ== 2022-03-07T17:43:01.384 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-07T17:43:01.384 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-07T17:43:01.385 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-07T17:43:01.409 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-07T17:43:02.201 INFO:teuthology.orchestra.run.smithi057.stdout:[client.b] 2022-03-07T17:43:02.201 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAmRCZiIWnXABAAV7QYesVgyAf6iK2Ge83zBg== 2022-03-07T17:43:02.225 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-07T17:43:02.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-07T17:43:02.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-07T17:43:02.248 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-07T17:43:02.979 INFO:teuthology.orchestra.run.smithi057.stdout:[client.c] 2022-03-07T17:43:02.980 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAmRCZi+ip0NBAATLq1sHAFxvkDNnzP4Kpnzw== 2022-03-07T17:43:03.004 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2022-03-07T17:43:03.004 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-07T17:43:03.004 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-07T17:43:03.027 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd dump --format=json 2022-03-07T17:43:03.551 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:03.552 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"fsid":"230d766b-6690-4686-b39d-409fa88d9c2b","created":"2022-03-07T17:38:08.372972+0000","modified":"2022-03-07T17:42:52.174293+0000","last_up_change":"2022-03-07T17:42:51.072503+0000","last_in_change":"2022-03-07T17:42:00.366446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-07T17:42:42.123973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a0a3f44c-d5bd-4d10-a153-dcce5ea96632","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6800","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6801","nonce":4087578219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6802","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6803","nonce":4087578219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6806","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6807","nonce":4087578219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6804","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6805","nonce":4087578219}]},"public_addr":"10.249.192.13:6801/4087578219","cluster_addr":"10.249.192.13:6803/4087578219","heartbeat_back_addr":"10.249.192.13:6807/4087578219","heartbeat_front_addr":"10.249.192.13:6805/4087578219","state":["exists","up"]},{"osd":1,"uuid":"fe993768-e368-4ca7-bff4-2526550590c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6800","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6801","nonce":2980916210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6802","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6803","nonce":2980916210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6806","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6807","nonce":2980916210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6804","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6805","nonce":2980916210}]},"public_addr":"10.249.192.15:6801/2980916210","cluster_addr":"10.249.192.15:6803/2980916210","heartbeat_back_addr":"10.249.192.15:6807/2980916210","heartbeat_front_addr":"10.249.192.15:6805/2980916210","state":["exists","up"]},{"osd":2,"uuid":"4cd9cb06-47ff-4a3c-9db5-f34a379447b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6800","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6801","nonce":546723948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6802","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6803","nonce":546723948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6806","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6807","nonce":546723948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6804","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6805","nonce":546723948}]},"public_addr":"10.249.192.14:6801/546723948","cluster_addr":"10.249.192.14:6803/546723948","heartbeat_back_addr":"10.249.192.14:6807/546723948","heartbeat_front_addr":"10.249.192.14:6805/546723948","state":["exists","up"]},{"osd":3,"uuid":"fba7297b-adf8-46ed-b7c6-ebab7e3fef3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6800","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6801","nonce":2864882251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6802","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6803","nonce":2864882251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6806","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6807","nonce":2864882251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6804","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6805","nonce":2864882251}]},"public_addr":"10.249.194.78:6801/2864882251","cluster_addr":"10.249.194.78:6803/2864882251","heartbeat_back_addr":"10.249.194.78:6807/2864882251","heartbeat_front_addr":"10.249.194.78:6805/2864882251","state":["exists","up"]},{"osd":4,"uuid":"11ecde4d-7e5e-463e-befc-f6d0e0c0c88a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6800","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6801","nonce":1062159791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6802","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6803","nonce":1062159791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6806","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6807","nonce":1062159791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6804","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6805","nonce":1062159791}]},"public_addr":"10.249.192.16:6801/1062159791","cluster_addr":"10.249.192.16:6803/1062159791","heartbeat_back_addr":"10.249.192.16:6807/1062159791","heartbeat_front_addr":"10.249.192.16:6805/1062159791","state":["exists","up"]},{"osd":5,"uuid":"d5fbf062-fcda-4737-a958-85f0ebc4c22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6800","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6801","nonce":2109684861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6802","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6803","nonce":2109684861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6806","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6807","nonce":2109684861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6804","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6805","nonce":2109684861}]},"public_addr":"10.249.194.80:6801/2109684861","cluster_addr":"10.249.194.80:6803/2109684861","heartbeat_back_addr":"10.249.194.80:6807/2109684861","heartbeat_front_addr":"10.249.194.80:6805/2109684861","state":["exists","up"]},{"osd":6,"uuid":"9c6ff0fa-2122-4d7f-95e7-0dda714b1100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6800","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6801","nonce":4207918039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6802","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6803","nonce":4207918039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6806","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6807","nonce":4207918039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6804","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6805","nonce":4207918039}]},"public_addr":"10.249.194.142:6801/4207918039","cluster_addr":"10.249.194.142:6803/4207918039","heartbeat_back_addr":"10.249.194.142:6807/4207918039","heartbeat_front_addr":"10.249.194.142:6805/4207918039","state":["exists","up"]},{"osd":7,"uuid":"c95efba4-516c-4295-8a02-d7eff80bcee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6800","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6801","nonce":3895179234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6802","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6803","nonce":3895179234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6806","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6807","nonce":3895179234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6804","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6805","nonce":3895179234}]},"public_addr":"10.249.194.141:6801/3895179234","cluster_addr":"10.249.194.141:6803/3895179234","heartbeat_back_addr":"10.249.194.141:6807/3895179234","heartbeat_front_addr":"10.249.194.141:6805/3895179234","state":["exists","up"]},{"osd":8,"uuid":"0ad4b55e-6d63-4126-9685-98fcf1a85145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6800","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6801","nonce":1389335553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6802","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6803","nonce":1389335553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6806","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6807","nonce":1389335553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6804","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6805","nonce":1389335553}]},"public_addr":"10.249.194.144:6801/1389335553","cluster_addr":"10.249.194.144:6803/1389335553","heartbeat_back_addr":"10.249.194.144:6807/1389335553","heartbeat_front_addr":"10.249.194.144:6805/1389335553","state":["exists","up"]},{"osd":9,"uuid":"991f9ab7-c5f9-471c-ac3e-adb23c8963d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6800","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6801","nonce":2197420006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6802","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6803","nonce":2197420006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6806","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6807","nonce":2197420006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6804","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6805","nonce":2197420006}]},"public_addr":"10.249.194.82:6801/2197420006","cluster_addr":"10.249.194.82:6803/2197420006","heartbeat_back_addr":"10.249.194.82:6807/2197420006","heartbeat_front_addr":"10.249.194.82:6805/2197420006","state":["exists","up"]},{"osd":10,"uuid":"cdc5f617-0186-47c8-9cde-cb9a69715263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6800","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6801","nonce":1261857633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6802","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6803","nonce":1261857633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6806","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6807","nonce":1261857633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6804","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6805","nonce":1261857633}]},"public_addr":"10.249.194.81:6801/1261857633","cluster_addr":"10.249.194.81:6803/1261857633","heartbeat_back_addr":"10.249.194.81:6807/1261857633","heartbeat_front_addr":"10.249.194.81:6805/1261857633","state":["exists","up"]},{"osd":11,"uuid":"58dd4eaf-e224-45d5-b80a-da9c7f27c019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6800","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6801","nonce":641602460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6802","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6803","nonce":641602460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6806","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6807","nonce":641602460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6804","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6805","nonce":641602460}]},"public_addr":"10.249.194.143:6801/641602460","cluster_addr":"10.249.194.143:6803/641602460","heartbeat_back_addr":"10.249.194.143:6807/641602460","heartbeat_front_addr":"10.249.194.143:6805/641602460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:35.553528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.515793+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.510277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:36.034562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.711278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:43.312726+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:41.503243+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:32.516807+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:41.975649+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:49.268326+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:46.532761+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.108750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-07T17:43:03.569 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-07T17:42:42.123973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-07T17:43:03.570 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd pool get .mgr pg_num 2022-03-07T17:43:04.093 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2022-03-07T17:43:04.117 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-07T17:43:04.118 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-07T17:43:04.118 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph mgr dump --format=json 2022-03-07T17:43:04.705 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:04.722 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"active_gid":24239,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.70:6800","nonce":1602155599},{"type":"v1","addr":"10.249.194.70:6801","nonce":1602155599}]},"active_addr":"10.249.194.70:6801/1602155599","active_change":"2022-03-07T17:40:44.680656+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.249.194.70:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.249.194.70:0","nonce":1913221783}]},{"addrvec":[{"type":"v2","addr":"10.249.194.70:0","nonce":30664115}]},{"addrvec":[{"type":"v2","addr":"10.249.194.70:0","nonce":522668611}]},{"addrvec":[{"type":"v2","addr":"10.249.194.70:0","nonce":2060500756}]}]}} 2022-03-07T17:43:04.726 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-07T17:43:04.727 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-07T17:43:04.727 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd dump --format=json 2022-03-07T17:43:05.265 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:05.265 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"fsid":"230d766b-6690-4686-b39d-409fa88d9c2b","created":"2022-03-07T17:38:08.372972+0000","modified":"2022-03-07T17:42:52.174293+0000","last_up_change":"2022-03-07T17:42:51.072503+0000","last_in_change":"2022-03-07T17:42:00.366446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-07T17:42:42.123973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a0a3f44c-d5bd-4d10-a153-dcce5ea96632","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6800","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6801","nonce":4087578219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6802","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6803","nonce":4087578219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6806","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6807","nonce":4087578219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6804","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6805","nonce":4087578219}]},"public_addr":"10.249.192.13:6801/4087578219","cluster_addr":"10.249.192.13:6803/4087578219","heartbeat_back_addr":"10.249.192.13:6807/4087578219","heartbeat_front_addr":"10.249.192.13:6805/4087578219","state":["exists","up"]},{"osd":1,"uuid":"fe993768-e368-4ca7-bff4-2526550590c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6800","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6801","nonce":2980916210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6802","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6803","nonce":2980916210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6806","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6807","nonce":2980916210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6804","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6805","nonce":2980916210}]},"public_addr":"10.249.192.15:6801/2980916210","cluster_addr":"10.249.192.15:6803/2980916210","heartbeat_back_addr":"10.249.192.15:6807/2980916210","heartbeat_front_addr":"10.249.192.15:6805/2980916210","state":["exists","up"]},{"osd":2,"uuid":"4cd9cb06-47ff-4a3c-9db5-f34a379447b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6800","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6801","nonce":546723948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6802","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6803","nonce":546723948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6806","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6807","nonce":546723948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6804","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6805","nonce":546723948}]},"public_addr":"10.249.192.14:6801/546723948","cluster_addr":"10.249.192.14:6803/546723948","heartbeat_back_addr":"10.249.192.14:6807/546723948","heartbeat_front_addr":"10.249.192.14:6805/546723948","state":["exists","up"]},{"osd":3,"uuid":"fba7297b-adf8-46ed-b7c6-ebab7e3fef3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6800","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6801","nonce":2864882251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6802","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6803","nonce":2864882251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6806","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6807","nonce":2864882251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6804","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6805","nonce":2864882251}]},"public_addr":"10.249.194.78:6801/2864882251","cluster_addr":"10.249.194.78:6803/2864882251","heartbeat_back_addr":"10.249.194.78:6807/2864882251","heartbeat_front_addr":"10.249.194.78:6805/2864882251","state":["exists","up"]},{"osd":4,"uuid":"11ecde4d-7e5e-463e-befc-f6d0e0c0c88a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6800","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6801","nonce":1062159791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6802","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6803","nonce":1062159791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6806","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6807","nonce":1062159791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6804","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6805","nonce":1062159791}]},"public_addr":"10.249.192.16:6801/1062159791","cluster_addr":"10.249.192.16:6803/1062159791","heartbeat_back_addr":"10.249.192.16:6807/1062159791","heartbeat_front_addr":"10.249.192.16:6805/1062159791","state":["exists","up"]},{"osd":5,"uuid":"d5fbf062-fcda-4737-a958-85f0ebc4c22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6800","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6801","nonce":2109684861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6802","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6803","nonce":2109684861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6806","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6807","nonce":2109684861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6804","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6805","nonce":2109684861}]},"public_addr":"10.249.194.80:6801/2109684861","cluster_addr":"10.249.194.80:6803/2109684861","heartbeat_back_addr":"10.249.194.80:6807/2109684861","heartbeat_front_addr":"10.249.194.80:6805/2109684861","state":["exists","up"]},{"osd":6,"uuid":"9c6ff0fa-2122-4d7f-95e7-0dda714b1100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6800","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6801","nonce":4207918039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6802","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6803","nonce":4207918039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6806","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6807","nonce":4207918039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6804","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6805","nonce":4207918039}]},"public_addr":"10.249.194.142:6801/4207918039","cluster_addr":"10.249.194.142:6803/4207918039","heartbeat_back_addr":"10.249.194.142:6807/4207918039","heartbeat_front_addr":"10.249.194.142:6805/4207918039","state":["exists","up"]},{"osd":7,"uuid":"c95efba4-516c-4295-8a02-d7eff80bcee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6800","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6801","nonce":3895179234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6802","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6803","nonce":3895179234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6806","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6807","nonce":3895179234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6804","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6805","nonce":3895179234}]},"public_addr":"10.249.194.141:6801/3895179234","cluster_addr":"10.249.194.141:6803/3895179234","heartbeat_back_addr":"10.249.194.141:6807/3895179234","heartbeat_front_addr":"10.249.194.141:6805/3895179234","state":["exists","up"]},{"osd":8,"uuid":"0ad4b55e-6d63-4126-9685-98fcf1a85145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6800","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6801","nonce":1389335553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6802","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6803","nonce":1389335553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6806","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6807","nonce":1389335553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6804","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6805","nonce":1389335553}]},"public_addr":"10.249.194.144:6801/1389335553","cluster_addr":"10.249.194.144:6803/1389335553","heartbeat_back_addr":"10.249.194.144:6807/1389335553","heartbeat_front_addr":"10.249.194.144:6805/1389335553","state":["exists","up"]},{"osd":9,"uuid":"991f9ab7-c5f9-471c-ac3e-adb23c8963d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6800","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6801","nonce":2197420006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6802","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6803","nonce":2197420006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6806","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6807","nonce":2197420006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6804","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6805","nonce":2197420006}]},"public_addr":"10.249.194.82:6801/2197420006","cluster_addr":"10.249.194.82:6803/2197420006","heartbeat_back_addr":"10.249.194.82:6807/2197420006","heartbeat_front_addr":"10.249.194.82:6805/2197420006","state":["exists","up"]},{"osd":10,"uuid":"cdc5f617-0186-47c8-9cde-cb9a69715263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6800","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6801","nonce":1261857633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6802","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6803","nonce":1261857633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6806","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6807","nonce":1261857633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6804","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6805","nonce":1261857633}]},"public_addr":"10.249.194.81:6801/1261857633","cluster_addr":"10.249.194.81:6803/1261857633","heartbeat_back_addr":"10.249.194.81:6807/1261857633","heartbeat_front_addr":"10.249.194.81:6805/1261857633","state":["exists","up"]},{"osd":11,"uuid":"58dd4eaf-e224-45d5-b80a-da9c7f27c019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6800","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6801","nonce":641602460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6802","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6803","nonce":641602460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6806","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6807","nonce":641602460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6804","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6805","nonce":641602460}]},"public_addr":"10.249.194.143:6801/641602460","cluster_addr":"10.249.194.143:6803/641602460","heartbeat_back_addr":"10.249.194.143:6807/641602460","heartbeat_front_addr":"10.249.194.143:6805/641602460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:35.553528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.515793+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.510277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:36.034562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.711278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:43.312726+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:41.503243+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:32.516807+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:41.975649+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:49.268326+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:46.532761+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.108750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-07T17:43:05.287 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-07T17:43:05.287 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd dump --format=json 2022-03-07T17:43:05.805 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:05.805 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":34,"fsid":"230d766b-6690-4686-b39d-409fa88d9c2b","created":"2022-03-07T17:38:08.372972+0000","modified":"2022-03-07T17:42:52.174293+0000","last_up_change":"2022-03-07T17:42:51.072503+0000","last_in_change":"2022-03-07T17:42:00.366446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-07T17:42:42.123973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a0a3f44c-d5bd-4d10-a153-dcce5ea96632","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6800","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6801","nonce":4087578219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6802","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6803","nonce":4087578219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6806","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6807","nonce":4087578219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.13:6804","nonce":4087578219},{"type":"v1","addr":"10.249.192.13:6805","nonce":4087578219}]},"public_addr":"10.249.192.13:6801/4087578219","cluster_addr":"10.249.192.13:6803/4087578219","heartbeat_back_addr":"10.249.192.13:6807/4087578219","heartbeat_front_addr":"10.249.192.13:6805/4087578219","state":["exists","up"]},{"osd":1,"uuid":"fe993768-e368-4ca7-bff4-2526550590c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6800","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6801","nonce":2980916210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6802","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6803","nonce":2980916210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6806","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6807","nonce":2980916210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.15:6804","nonce":2980916210},{"type":"v1","addr":"10.249.192.15:6805","nonce":2980916210}]},"public_addr":"10.249.192.15:6801/2980916210","cluster_addr":"10.249.192.15:6803/2980916210","heartbeat_back_addr":"10.249.192.15:6807/2980916210","heartbeat_front_addr":"10.249.192.15:6805/2980916210","state":["exists","up"]},{"osd":2,"uuid":"4cd9cb06-47ff-4a3c-9db5-f34a379447b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6800","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6801","nonce":546723948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6802","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6803","nonce":546723948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6806","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6807","nonce":546723948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.14:6804","nonce":546723948},{"type":"v1","addr":"10.249.192.14:6805","nonce":546723948}]},"public_addr":"10.249.192.14:6801/546723948","cluster_addr":"10.249.192.14:6803/546723948","heartbeat_back_addr":"10.249.192.14:6807/546723948","heartbeat_front_addr":"10.249.192.14:6805/546723948","state":["exists","up"]},{"osd":3,"uuid":"fba7297b-adf8-46ed-b7c6-ebab7e3fef3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6800","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6801","nonce":2864882251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6802","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6803","nonce":2864882251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6806","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6807","nonce":2864882251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.78:6804","nonce":2864882251},{"type":"v1","addr":"10.249.194.78:6805","nonce":2864882251}]},"public_addr":"10.249.194.78:6801/2864882251","cluster_addr":"10.249.194.78:6803/2864882251","heartbeat_back_addr":"10.249.194.78:6807/2864882251","heartbeat_front_addr":"10.249.194.78:6805/2864882251","state":["exists","up"]},{"osd":4,"uuid":"11ecde4d-7e5e-463e-befc-f6d0e0c0c88a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6800","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6801","nonce":1062159791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6802","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6803","nonce":1062159791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6806","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6807","nonce":1062159791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.192.16:6804","nonce":1062159791},{"type":"v1","addr":"10.249.192.16:6805","nonce":1062159791}]},"public_addr":"10.249.192.16:6801/1062159791","cluster_addr":"10.249.192.16:6803/1062159791","heartbeat_back_addr":"10.249.192.16:6807/1062159791","heartbeat_front_addr":"10.249.192.16:6805/1062159791","state":["exists","up"]},{"osd":5,"uuid":"d5fbf062-fcda-4737-a958-85f0ebc4c22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6800","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6801","nonce":2109684861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6802","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6803","nonce":2109684861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6806","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6807","nonce":2109684861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.80:6804","nonce":2109684861},{"type":"v1","addr":"10.249.194.80:6805","nonce":2109684861}]},"public_addr":"10.249.194.80:6801/2109684861","cluster_addr":"10.249.194.80:6803/2109684861","heartbeat_back_addr":"10.249.194.80:6807/2109684861","heartbeat_front_addr":"10.249.194.80:6805/2109684861","state":["exists","up"]},{"osd":6,"uuid":"9c6ff0fa-2122-4d7f-95e7-0dda714b1100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6800","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6801","nonce":4207918039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6802","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6803","nonce":4207918039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6806","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6807","nonce":4207918039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.142:6804","nonce":4207918039},{"type":"v1","addr":"10.249.194.142:6805","nonce":4207918039}]},"public_addr":"10.249.194.142:6801/4207918039","cluster_addr":"10.249.194.142:6803/4207918039","heartbeat_back_addr":"10.249.194.142:6807/4207918039","heartbeat_front_addr":"10.249.194.142:6805/4207918039","state":["exists","up"]},{"osd":7,"uuid":"c95efba4-516c-4295-8a02-d7eff80bcee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6800","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6801","nonce":3895179234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6802","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6803","nonce":3895179234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6806","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6807","nonce":3895179234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.141:6804","nonce":3895179234},{"type":"v1","addr":"10.249.194.141:6805","nonce":3895179234}]},"public_addr":"10.249.194.141:6801/3895179234","cluster_addr":"10.249.194.141:6803/3895179234","heartbeat_back_addr":"10.249.194.141:6807/3895179234","heartbeat_front_addr":"10.249.194.141:6805/3895179234","state":["exists","up"]},{"osd":8,"uuid":"0ad4b55e-6d63-4126-9685-98fcf1a85145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6800","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6801","nonce":1389335553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6802","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6803","nonce":1389335553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6806","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6807","nonce":1389335553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.144:6804","nonce":1389335553},{"type":"v1","addr":"10.249.194.144:6805","nonce":1389335553}]},"public_addr":"10.249.194.144:6801/1389335553","cluster_addr":"10.249.194.144:6803/1389335553","heartbeat_back_addr":"10.249.194.144:6807/1389335553","heartbeat_front_addr":"10.249.194.144:6805/1389335553","state":["exists","up"]},{"osd":9,"uuid":"991f9ab7-c5f9-471c-ac3e-adb23c8963d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6800","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6801","nonce":2197420006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6802","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6803","nonce":2197420006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6806","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6807","nonce":2197420006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.82:6804","nonce":2197420006},{"type":"v1","addr":"10.249.194.82:6805","nonce":2197420006}]},"public_addr":"10.249.194.82:6801/2197420006","cluster_addr":"10.249.194.82:6803/2197420006","heartbeat_back_addr":"10.249.194.82:6807/2197420006","heartbeat_front_addr":"10.249.194.82:6805/2197420006","state":["exists","up"]},{"osd":10,"uuid":"cdc5f617-0186-47c8-9cde-cb9a69715263","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6800","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6801","nonce":1261857633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6802","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6803","nonce":1261857633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6806","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6807","nonce":1261857633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.81:6804","nonce":1261857633},{"type":"v1","addr":"10.249.194.81:6805","nonce":1261857633}]},"public_addr":"10.249.194.81:6801/1261857633","cluster_addr":"10.249.194.81:6803/1261857633","heartbeat_back_addr":"10.249.194.81:6807/1261857633","heartbeat_front_addr":"10.249.194.81:6805/1261857633","state":["exists","up"]},{"osd":11,"uuid":"58dd4eaf-e224-45d5-b80a-da9c7f27c019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6800","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6801","nonce":641602460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6802","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6803","nonce":641602460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6806","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6807","nonce":641602460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.194.143:6804","nonce":641602460},{"type":"v1","addr":"10.249.194.143:6805","nonce":641602460}]},"public_addr":"10.249.194.143:6801/641602460","cluster_addr":"10.249.194.143:6803/641602460","heartbeat_back_addr":"10.249.194.143:6807/641602460","heartbeat_front_addr":"10.249.194.143:6805/641602460","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:35.553528+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.515793+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.510277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:36.034562+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.711278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:43.312726+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:41.503243+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:32.516807+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:41.975649+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:49.268326+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:46.532761+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T17:42:40.108750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-07T17:43:05.825 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.0 flush_pg_stats 2022-03-07T17:43:05.825 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.1 flush_pg_stats 2022-03-07T17:43:05.826 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.2 flush_pg_stats 2022-03-07T17:43:05.827 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.3 flush_pg_stats 2022-03-07T17:43:05.828 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.4 flush_pg_stats 2022-03-07T17:43:05.829 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.5 flush_pg_stats 2022-03-07T17:43:05.829 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.6 flush_pg_stats 2022-03-07T17:43:05.830 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.7 flush_pg_stats 2022-03-07T17:43:05.831 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.8 flush_pg_stats 2022-03-07T17:43:05.831 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.9 flush_pg_stats 2022-03-07T17:43:05.832 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.10 flush_pg_stats 2022-03-07T17:43:05.832 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph tell osd.11 flush_pg_stats 2022-03-07T17:43:06.406 INFO:teuthology.orchestra.run.smithi057.stdout:111669149702 2022-03-07T17:43:06.406 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.1 2022-03-07T17:43:06.724 INFO:teuthology.orchestra.run.smithi057.stdout:111669149702 2022-03-07T17:43:06.725 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.2 2022-03-07T17:43:06.731 INFO:teuthology.orchestra.run.smithi057.stdout:103079215112 2022-03-07T17:43:06.731 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.3 2022-03-07T17:43:06.736 INFO:teuthology.orchestra.run.smithi057.stdout:141733920772 2022-03-07T17:43:06.737 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.9 2022-03-07T17:43:06.738 INFO:teuthology.orchestra.run.smithi057.stdout:103079215113 2022-03-07T17:43:06.739 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.0 2022-03-07T17:43:06.745 INFO:teuthology.orchestra.run.smithi057.stdout:90194313223 2022-03-07T17:43:06.745 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.7 2022-03-07T17:43:06.749 INFO:teuthology.orchestra.run.smithi057.stdout:115964116998 2022-03-07T17:43:06.750 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.6 2022-03-07T17:43:06.753 INFO:teuthology.orchestra.run.smithi057.stdout:111669149704 2022-03-07T17:43:06.754 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.11 2022-03-07T17:43:06.885 INFO:teuthology.orchestra.run.smithi057.stdout:115964116999 2022-03-07T17:43:06.886 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.8 2022-03-07T17:43:06.982 INFO:teuthology.orchestra.run.smithi057.stdout:120259084294 2022-03-07T17:43:06.983 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.5 2022-03-07T17:43:06.998 INFO:teuthology.orchestra.run.smithi057.stdout:133143986183 2022-03-07T17:43:06.999 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.10 2022-03-07T17:43:07.105 INFO:teuthology.orchestra.run.smithi057.stdout:111669149703 2022-03-07T17:43:07.105 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.4 2022-03-07T17:43:07.367 INFO:teuthology.orchestra.run.smithi057.stdout:111669149701 2022-03-07T17:43:07.497 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149701 for osd.1 2022-03-07T17:43:07.587 INFO:teuthology.orchestra.run.smithi057.stdout:103079215111 2022-03-07T17:43:07.727 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215112 got 103079215111 for osd.3 2022-03-07T17:43:07.832 INFO:teuthology.orchestra.run.smithi057.stdout:111669149701 2022-03-07T17:43:07.839 INFO:teuthology.orchestra.run.smithi057.stdout:103079215112 2022-03-07T17:43:07.867 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149701 for osd.2 2022-03-07T17:43:07.874 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215113 got 103079215112 for osd.0 2022-03-07T17:43:07.885 INFO:teuthology.orchestra.run.smithi057.stdout:111669149704 2022-03-07T17:43:07.900 INFO:teuthology.orchestra.run.smithi057.stdout:141733920773 2022-03-07T17:43:07.912 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149704 got 111669149704 for osd.11 2022-03-07T17:43:07.912 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:07.931 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920772 got 141733920773 for osd.9 2022-03-07T17:43:07.933 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.064 INFO:teuthology.orchestra.run.smithi057.stdout:115964116998 2022-03-07T17:43:08.089 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116998 for osd.6 2022-03-07T17:43:08.090 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.124 INFO:teuthology.orchestra.run.smithi057.stdout:115964116999 2022-03-07T17:43:08.149 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116999 for osd.8 2022-03-07T17:43:08.149 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.158 INFO:teuthology.orchestra.run.smithi057.stdout:90194313224 2022-03-07T17:43:08.177 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313223 got 90194313224 for osd.7 2022-03-07T17:43:08.177 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.227 INFO:teuthology.orchestra.run.smithi057.stdout:111669149703 2022-03-07T17:43:08.248 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149703 for osd.4 2022-03-07T17:43:08.248 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:120259084294 2022-03-07T17:43:08.322 INFO:teuthology.orchestra.run.smithi057.stdout:133143986183 2022-03-07T17:43:08.327 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084294 got 120259084294 for osd.5 2022-03-07T17:43:08.328 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.340 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986183 got 133143986183 for osd.10 2022-03-07T17:43:08.341 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:08.498 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.1 2022-03-07T17:43:08.729 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.3 2022-03-07T17:43:08.868 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.2 2022-03-07T17:43:08.876 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph osd last-stat-seq osd.0 2022-03-07T17:43:09.118 INFO:teuthology.orchestra.run.smithi057.stdout:111669149702 2022-03-07T17:43:09.140 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149702 for osd.1 2022-03-07T17:43:09.140 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:09.402 INFO:teuthology.orchestra.run.smithi057.stdout:103079215113 2022-03-07T17:43:09.409 INFO:teuthology.orchestra.run.smithi057.stdout:111669149702 2022-03-07T17:43:09.430 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215112 got 103079215113 for osd.3 2022-03-07T17:43:09.430 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:09.434 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149702 for osd.2 2022-03-07T17:43:09.435 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:103079215114 2022-03-07T17:43:09.549 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215113 got 103079215114 for osd.0 2022-03-07T17:43:09.550 DEBUG:teuthology.parallel:result is None 2022-03-07T17:43:09.550 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-07T17:43:09.550 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph pg dump --format=json 2022-03-07T17:43:10.080 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:10.082 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2022-03-07T17:43:10.102 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2022-03-07T17:43:09.526326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65860,"kb_used_data":5348,"kb_used_omap":0,"kb_used_meta":60032,"kb_avail":1124875964,"statfs":{"total":1151940427776,"available":1151872987136,"internally_reserved":0,"allocated":5476352,"data_stored":3048975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61472768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":36,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004194"},"pg_stats":[{"pgid":"1.0","version":"34'137","reported_seq":195,"reported_epoch":34,"state":"active+clean","last_fresh":"2022-03-07T17:42:59.708204+0000","last_change":"2022-03-07T17:42:47.543355+0000","last_active":"2022-03-07T17:42:59.708204+0000","last_peered":"2022-03-07T17:42:59.708204+0000","last_clean":"2022-03-07T17:42:59.708204+0000","last_became_active":"2022-03-07T17:42:47.541811+0000","last_became_peered":"2022-03-07T17:42:47.541811+0000","last_unstale":"2022-03-07T17:42:59.708204+0000","last_undegraded":"2022-03-07T17:42:59.708204+0000","last_fullsized":"2022-03-07T17:42:59.708204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-07T17:42:42.612437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-07T17:42:42.612437+0000","last_clean_scrub_stamp":"2022-03-07T17:42:42.612437+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-08T22:23:56.213136+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":9,"up_from":33,"seq":141733920773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.663}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9990000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3690000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2679999999999998}]}]},{"osd":10,"up_from":31,"seq":133143986183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2389999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.012}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9750000000000001}]}]},{"osd":5,"up_from":28,"seq":120259084294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":856064,"data_stored":651001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.615}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.02}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.96}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0710000000000002}]}]},{"osd":8,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2629999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7070000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.121}]}]},{"osd":6,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8320000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":11,"up_from":26,"seq":111669149705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8779999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3069999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.149}]}]},{"osd":0,"up_from":24,"seq":103079215114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.329}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0539999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2890000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3399999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4129999999999998}]}]},{"osd":3,"up_from":24,"seq":103079215113,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5184,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739968,"statfs":{"total":95995035648,"available":95989727232,"internally_reserved":0,"allocated":319488,"data_stored":119388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0380000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":1,"up_from":26,"seq":111669149703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":856064,"data_stored":651001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0359999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.169}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.929}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.065}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.411}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.441000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.956}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.462999999999999}]}]},{"osd":2,"up_from":26,"seq":111669149703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":856064,"data_stored":651001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":4,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.619}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.411}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4369999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3420000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2280000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-07T17:43:10.105 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph pg dump --format=json 2022-03-07T17:43:10.669 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2022-03-07T17:43:10.686 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2022-03-07T17:43:09.526326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65860,"kb_used_data":5348,"kb_used_omap":0,"kb_used_meta":60032,"kb_avail":1124875964,"statfs":{"total":1151940427776,"available":1151872987136,"internally_reserved":0,"allocated":5476352,"data_stored":3048975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61472768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":36,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004194"},"pg_stats":[{"pgid":"1.0","version":"34'137","reported_seq":195,"reported_epoch":34,"state":"active+clean","last_fresh":"2022-03-07T17:42:59.708204+0000","last_change":"2022-03-07T17:42:47.543355+0000","last_active":"2022-03-07T17:42:59.708204+0000","last_peered":"2022-03-07T17:42:59.708204+0000","last_clean":"2022-03-07T17:42:59.708204+0000","last_became_active":"2022-03-07T17:42:47.541811+0000","last_became_peered":"2022-03-07T17:42:47.541811+0000","last_unstale":"2022-03-07T17:42:59.708204+0000","last_undegraded":"2022-03-07T17:42:59.708204+0000","last_fullsized":"2022-03-07T17:42:59.708204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-07T17:42:42.612437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-07T17:42:42.612437+0000","last_clean_scrub_stamp":"2022-03-07T17:42:42.612437+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-08T22:23:56.213136+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":9,"up_from":33,"seq":141733920773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.663}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9990000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3690000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2679999999999998}]}]},{"osd":10,"up_from":31,"seq":133143986183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2389999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.012}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9750000000000001}]}]},{"osd":5,"up_from":28,"seq":120259084294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":856064,"data_stored":651001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.615}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.02}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.96}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0710000000000002}]}]},{"osd":8,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2629999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7070000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.121}]}]},{"osd":6,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8320000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":11,"up_from":26,"seq":111669149705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8779999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0630000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3069999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.149}]}]},{"osd":0,"up_from":24,"seq":103079215114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.329}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0539999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2890000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3399999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4129999999999998}]}]},{"osd":3,"up_from":24,"seq":103079215113,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5184,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739968,"statfs":{"total":95995035648,"available":95989727232,"internally_reserved":0,"allocated":319488,"data_stored":119388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0380000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":1,"up_from":26,"seq":111669149703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":856064,"data_stored":651001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0359999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.169}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.929}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.065}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.411}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.441000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.956}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.462999999999999}]}]},{"osd":2,"up_from":26,"seq":111669149703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6412,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738740,"statfs":{"total":95995035648,"available":95988469760,"internally_reserved":0,"allocated":856064,"data_stored":651001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8069999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":4,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":122073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.619}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.411}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4369999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3420000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2280000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-07T17:43:10.688 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-07T17:43:10.688 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-07T17:43:10.689 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-07T17:43:10.689 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph health --format=json 2022-03-07T17:43:11.271 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:11.272 INFO:teuthology.orchestra.run.smithi057.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-07T17:43:11.296 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-07T17:43:11.296 INFO:tasks.rook:Rook complete, yielding 2022-03-07T17:43:11.296 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-07T17:43:11.308 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph -s 2022-03-07T17:43:11.910 INFO:teuthology.orchestra.run.smithi057.stdout: cluster: 2022-03-07T17:43:11.911 INFO:teuthology.orchestra.run.smithi057.stdout: id: 230d766b-6690-4686-b39d-409fa88d9c2b 2022-03-07T17:43:11.911 INFO:teuthology.orchestra.run.smithi057.stdout: health: HEALTH_OK 2022-03-07T17:43:11.911 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:11.911 INFO:teuthology.orchestra.run.smithi057.stdout: services: 2022-03-07T17:43:11.912 INFO:teuthology.orchestra.run.smithi057.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-07T17:43:11.912 INFO:teuthology.orchestra.run.smithi057.stdout: mgr: a(active, since 2m) 2022-03-07T17:43:11.912 INFO:teuthology.orchestra.run.smithi057.stdout: osd: 12 osds: 12 up (since 20s), 12 in (since 71s) 2022-03-07T17:43:11.912 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:11.913 INFO:teuthology.orchestra.run.smithi057.stdout: data: 2022-03-07T17:43:11.913 INFO:teuthology.orchestra.run.smithi057.stdout: pools: 1 pools, 1 pgs 2022-03-07T17:43:11.913 INFO:teuthology.orchestra.run.smithi057.stdout: objects: 2 objects, 577 KiB 2022-03-07T17:43:11.913 INFO:teuthology.orchestra.run.smithi057.stdout: usage: 64 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-07T17:43:11.913 INFO:teuthology.orchestra.run.smithi057.stdout: pgs: 1 active+clean 2022-03-07T17:43:11.914 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:11.914 INFO:teuthology.orchestra.run.smithi057.stdout: io: 2022-03-07T17:43:11.914 INFO:teuthology.orchestra.run.smithi057.stdout: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr 2022-03-07T17:43:11.914 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:43:11.930 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status 2022-03-07T17:43:12.643 INFO:teuthology.orchestra.run.smithi057.stdout:Backend: rook 2022-03-07T17:43:12.644 INFO:teuthology.orchestra.run.smithi057.stdout:Available: Yes 2022-03-07T17:43:12.670 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ps 2022-03-07T17:43:13.259 INFO:teuthology.orchestra.run.smithi057.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-07T17:43:13.260 INFO:teuthology.orchestra.run.smithi057.stdout:crashcollector.crash smithi057 running (2m) 0s ago 2m - - 02e0d18029b5 2022-03-07T17:43:13.261 INFO:teuthology.orchestra.run.smithi057.stdout:crashcollector.crash smithi066 running (71s) 0s ago 71s - - 02e0d18029b5 2022-03-07T17:43:13.261 INFO:teuthology.orchestra.run.smithi057.stdout:crashcollector.crash smithi085 running (2m) 0s ago 2m - - 02e0d18029b5 2022-03-07T17:43:13.261 INFO:teuthology.orchestra.run.smithi057.stdout:mgr.a smithi066 running (2m) 0s ago 2m - - 02e0d18029b5 2022-03-07T17:43:13.261 INFO:teuthology.orchestra.run.smithi057.stdout:mon.a smithi085 running (5m) 0s ago 5m - - 02e0d18029b5 2022-03-07T17:43:13.261 INFO:teuthology.orchestra.run.smithi057.stdout:mon.b smithi057 running (5m) 0s ago 5m - - 02e0d18029b5 2022-03-07T17:43:13.262 INFO:teuthology.orchestra.run.smithi057.stdout:mon.c smithi066 running (4m) 0s ago 4m - - 02e0d18029b5 2022-03-07T17:43:13.262 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 smithi057 running (76s) 0s ago 76s - - 02e0d18029b5 2022-03-07T17:43:13.262 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 smithi057 running (73s) 0s ago 73s - - 02e0d18029b5 2022-03-07T17:43:13.262 INFO:teuthology.orchestra.run.smithi057.stdout:osd.10 smithi066 running (61s) 0s ago 61s - - 02e0d18029b5 2022-03-07T17:43:13.263 INFO:teuthology.orchestra.run.smithi057.stdout:osd.11 smithi085 running (63s) 0s ago 63s - - 02e0d18029b5 2022-03-07T17:43:13.263 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 smithi057 running (74s) 0s ago 74s - - 02e0d18029b5 2022-03-07T17:43:13.263 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 smithi066 running (71s) 0s ago 71s - - 02e0d18029b5 2022-03-07T17:43:13.263 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 smithi057 running (70s) 0s ago 70s - - 02e0d18029b5 2022-03-07T17:43:13.264 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 smithi066 running (68s) 0s ago 69s - - 02e0d18029b5 2022-03-07T17:43:13.264 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 smithi085 running (65s) 0s ago 66s - - 02e0d18029b5 2022-03-07T17:43:13.265 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 smithi085 running (66s) 0s ago 67s - - 02e0d18029b5 2022-03-07T17:43:13.265 INFO:teuthology.orchestra.run.smithi057.stdout:osd.8 smithi085 running (64s) 0s ago 64s - - 02e0d18029b5 2022-03-07T17:43:13.265 INFO:teuthology.orchestra.run.smithi057.stdout:osd.9 smithi066 running (59s) 0s ago 60s - - 02e0d18029b5 2022-03-07T17:43:13.284 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls 2022-03-07T17:43:13.957 INFO:teuthology.orchestra.run.smithi057.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-07T17:43:13.958 INFO:teuthology.orchestra.run.smithi057.stdout:crash 3/3 0s ago 2m * 2022-03-07T17:43:13.958 INFO:teuthology.orchestra.run.smithi057.stdout:mgr 1/1 0s ago 2m count:1 2022-03-07T17:43:13.958 INFO:teuthology.orchestra.run.smithi057.stdout:mon 3/3 0s ago 5m count:3 2022-03-07T17:43:13.959 INFO:teuthology.orchestra.run.smithi057.stdout:osd 24 0s ago 76s 2022-03-07T17:43:13.959 INFO:teuthology.orchestra.run.smithi057.stdout:osd.all-available-devices 0 0s ago - * 2022-03-07T17:43:13.987 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch device ls 2022-03-07T17:43:14.556 INFO:teuthology.orchestra.run.smithi057.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-07T17:43:14.556 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T17:43:14.556 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T17:43:14.557 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T17:43:14.557 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T17:43:14.557 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T17:43:14.557 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T17:43:14.558 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T17:43:14.558 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T17:43:14.558 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T17:43:14.558 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T17:43:14.558 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T17:43:14.559 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T17:43:14.581 INFO:teuthology.run_tasks:Running task exec... 2022-03-07T17:43:14.593 INFO:teuthology.task.exec:Executing custom commands... 2022-03-07T17:43:14.594 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi057.front.sepia.ceph.com 2022-03-07T17:43:14.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-07T17:43:14.595 DEBUG:teuthology.orchestra.run.smithi057:> toolbox() { 2022-03-07T17:43:14.595 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-07T17:43:14.595 DEBUG:teuthology.orchestra.run.smithi057:> } 2022-03-07T17:43:14.595 DEBUG:teuthology.orchestra.run.smithi057:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-07T17:43:14.596 DEBUG:teuthology.orchestra.run.smithi057:> toolbox ceph orch osd rm 0 --force 2022-03-07T17:43:14.596 DEBUG:teuthology.orchestra.run.smithi057:> removed_pv="" 2022-03-07T17:43:14.596 DEBUG:teuthology.orchestra.run.smithi057:> while [ "$removed_pv" = "" ] 2022-03-07T17:43:14.597 DEBUG:teuthology.orchestra.run.smithi057:> do 2022-03-07T17:43:14.597 DEBUG:teuthology.orchestra.run.smithi057:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-07T17:43:14.597 DEBUG:teuthology.orchestra.run.smithi057:> sleep 3s 2022-03-07T17:43:14.598 DEBUG:teuthology.orchestra.run.smithi057:> done 2022-03-07T17:43:14.598 DEBUG:teuthology.orchestra.run.smithi057:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-07T17:43:14.598 DEBUG:teuthology.orchestra.run.smithi057:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-07T17:43:14.598 DEBUG:teuthology.orchestra.run.smithi057:> toolbox ceph orch device zap $host $target_path --force 2022-03-07T17:43:14.599 DEBUG:teuthology.orchestra.run.smithi057:> zap_completion="0" 2022-03-07T17:43:14.599 DEBUG:teuthology.orchestra.run.smithi057:> while [ "$zap_completion" = "0" ] 2022-03-07T17:43:14.599 DEBUG:teuthology.orchestra.run.smithi057:> do 2022-03-07T17:43:14.600 DEBUG:teuthology.orchestra.run.smithi057:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-07T17:43:14.600 DEBUG:teuthology.orchestra.run.smithi057:> sleep 3s 2022-03-07T17:43:14.600 DEBUG:teuthology.orchestra.run.smithi057:> done 2022-03-07T17:43:14.601 DEBUG:teuthology.orchestra.run.smithi057:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-07T17:43:14.601 DEBUG:teuthology.orchestra.run.smithi057:> toolbox ceph orch apply osd --all-available-devices 2022-03-07T17:43:14.601 DEBUG:teuthology.orchestra.run.smithi057:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-07T17:43:14.602 DEBUG:teuthology.orchestra.run.smithi057:> num_osd="0" 2022-03-07T17:43:14.602 DEBUG:teuthology.orchestra.run.smithi057:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-07T17:43:14.602 DEBUG:teuthology.orchestra.run.smithi057:> do 2022-03-07T17:43:14.602 DEBUG:teuthology.orchestra.run.smithi057:> echo "waiting for osd to come back up" 2022-03-07T17:43:14.603 DEBUG:teuthology.orchestra.run.smithi057:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-07T17:43:14.603 DEBUG:teuthology.orchestra.run.smithi057:> sleep 30s 2022-03-07T17:43:14.603 DEBUG:teuthology.orchestra.run.smithi057:> done 2022-03-07T17:43:14.604 DEBUG:teuthology.orchestra.run.smithi057:> ' 2022-03-07T17:43:14.619 INFO:teuthology.orchestra.run.smithi057.stderr:++ toolbox ceph osd stat 2022-03-07T17:43:14.620 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f3 -d ' ' 2022-03-07T17:43:14.620 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T17:43:17.029 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:43:17.512 INFO:teuthology.orchestra.run.smithi057.stderr:+ orig_num_osd=12 2022-03-07T17:43:17.513 INFO:teuthology.orchestra.run.smithi057.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-07T17:43:17.513 INFO:teuthology.orchestra.run.smithi057.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-07T17:43:17.580 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:43:18.232 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:18.222552 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-07T17:43:18.232 INFO:tasks.rook.operator.smithi057.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", AllowUnsupported: true}, 2022-03-07T17:43:18.232 INFO:tasks.rook.operator.smithi057.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-07T17:43:18.233 INFO:tasks.rook.operator.smithi057.stdout:   ... // 3 identical fields 2022-03-07T17:43:18.233 INFO:tasks.rook.operator.smithi057.stdout:   Config: nil, 2022-03-07T17:43:18.233 INFO:tasks.rook.operator.smithi057.stdout:   Selection: {}, 2022-03-07T17:43:18.233 INFO:tasks.rook.operator.smithi057.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-07T17:43:18.234 INFO:tasks.rook.operator.smithi057.stdout:-  { 2022-03-07T17:43:18.234 INFO:tasks.rook.operator.smithi057.stdout:-  Name: "smithi057-lv-1", 2022-03-07T17:43:18.234 INFO:tasks.rook.operator.smithi057.stdout:-  Count: 1, 2022-03-07T17:43:18.234 INFO:tasks.rook.operator.smithi057.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-07T17:43:18.234 INFO:tasks.rook.operator.smithi057.stdout:-  }, 2022-03-07T17:43:18.235 INFO:tasks.rook.operator.smithi057.stdout:   {Name: "smithi057-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi057-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-07T17:43:18.235 INFO:tasks.rook.operator.smithi057.stdout:   {Name: "smithi057-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi057-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-07T17:43:18.235 INFO:tasks.rook.operator.smithi057.stdout:   ... // 9 identical elements 2022-03-07T17:43:18.235 INFO:tasks.rook.operator.smithi057.stdout:   }, 2022-03-07T17:43:18.236 INFO:tasks.rook.operator.smithi057.stdout:   }, 2022-03-07T17:43:18.236 INFO:tasks.rook.operator.smithi057.stdout:   Annotations: nil, 2022-03-07T17:43:18.236 INFO:tasks.rook.operator.smithi057.stdout:   Labels: nil, 2022-03-07T17:43:18.236 INFO:tasks.rook.operator.smithi057.stdout:   ... // 20 identical fields 2022-03-07T17:43:18.236 INFO:tasks.rook.operator.smithi057.stdout:  } 2022-03-07T17:43:18.237 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:18.222640 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:43:18.338 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:18.333701 I | op-mon: parsing mon endpoints: a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 2022-03-07T17:43:18.493 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:18.487913 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:43:21.799 INFO:teuthology.orchestra.run.smithi057.stdout:Success 2022-03-07T17:43:21.827 INFO:teuthology.orchestra.run.smithi057.stderr:+ removed_pv= 2022-03-07T17:43:21.827 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' '' = '' ']' 2022-03-07T17:43:21.828 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl get pv 2022-03-07T17:43:21.828 INFO:teuthology.orchestra.run.smithi057.stderr:++ grep Released 2022-03-07T17:43:21.828 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f1 -d ' ' 2022-03-07T17:43:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:+ removed_pv= 2022-03-07T17:43:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 3s 2022-03-07T17:43:22.853 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:22.848792 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-07T17:43:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' '' = '' ']' 2022-03-07T17:43:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl get pv 2022-03-07T17:43:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:++ grep Released 2022-03-07T17:43:24.890 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f1 -d ' ' 2022-03-07T17:43:24.956 INFO:teuthology.orchestra.run.smithi057.stderr:+ removed_pv=smithi057-lv-1 2022-03-07T17:43:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 3s 2022-03-07T17:43:25.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:25.819924 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:43:25.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:25.819962 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:43:25.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:25.819983 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:43:25.832 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:25.828746 I | op-mon: parsing mon endpoints: a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 2022-03-07T17:43:25.875 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:25.869559 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:25.875 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:25.869851 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:26.281 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:26.277350 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:43:26.382 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:26.377885 I | op-mon: start running mons 2022-03-07T17:43:26.402 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:26.397944 I | op-mon: parsing mon endpoints: a=10.102.1.250:6789,b=10.103.120.199:6789,c=10.108.168.68:6789 2022-03-07T17:43:26.468 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:26.464262 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.120.199:6789","10.108.168.68:6789","10.102.1.250:6789"]}] data:c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:43:26.609 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:26.605025 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:26.610 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:26.605353 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:27.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:27.763518 I | op-mon: targeting the mon count 3 2022-03-07T17:43:27.788 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:27.783793 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-07T17:43:27.788 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:27.783956 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:43:27.960 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' smithi057-lv-1 = '' ']' 2022-03-07T17:43:27.960 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl get pv smithi057-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-07T17:43:28.022 INFO:teuthology.orchestra.run.smithi057.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-07T17:43:28.023 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f1 -d - 2022-03-07T17:43:28.023 INFO:teuthology.orchestra.run.smithi057.stderr:++ echo smithi057-lv-1 2022-03-07T17:43:28.024 INFO:teuthology.orchestra.run.smithi057.stderr:+ host=smithi057 2022-03-07T17:43:28.024 INFO:teuthology.orchestra.run.smithi057.stderr:+ toolbox ceph orch device zap smithi057 /dev/vg_nvme/lv_1 --force 2022-03-07T17:43:28.024 INFO:teuthology.orchestra.run.smithi057.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi057 /dev/vg_nvme/lv_1 --force 2022-03-07T17:43:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:43:28.168 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:28.164252 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:43:28.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:28.182887 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:43:28.384 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:28.379395 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-07T17:43:28.389 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:28.384996 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T17:43:28.609 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_1 on smithi057 zapped 2022-03-07T17:43:28.632 INFO:teuthology.orchestra.run.smithi057.stderr:+ zap_completion=0 2022-03-07T17:43:28.633 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' 0 = 0 ']' 2022-03-07T17:43:28.633 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-07T17:43:28.699 INFO:teuthology.orchestra.run.smithi057.stderr:+ zap_completion= 2022-03-07T17:43:28.700 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 3s 2022-03-07T17:43:30.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:30.393591 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-07T17:43:31.701 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' '' = 0 ']' 2022-03-07T17:43:31.701 INFO:teuthology.orchestra.run.smithi057.stderr:+ kubectl patch pv smithi057-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-07T17:43:31.784 INFO:teuthology.orchestra.run.smithi057.stdout:persistentvolume/smithi057-lv-1 patched 2022-03-07T17:43:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-07T17:43:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-07T17:43:31.851 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:43:32.719 INFO:teuthology.orchestra.run.smithi057.stdout:['Success'] 2022-03-07T17:43:32.731 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:32.715164 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-07T17:43:32.731 INFO:tasks.rook.operator.smithi057.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", AllowUnsupported: true}, 2022-03-07T17:43:32.732 INFO:tasks.rook.operator.smithi057.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-07T17:43:32.732 INFO:tasks.rook.operator.smithi057.stdout:   ... // 3 identical fields 2022-03-07T17:43:32.732 INFO:tasks.rook.operator.smithi057.stdout:   Config: nil, 2022-03-07T17:43:32.732 INFO:tasks.rook.operator.smithi057.stdout:   Selection: {}, 2022-03-07T17:43:32.732 INFO:tasks.rook.operator.smithi057.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-07T17:43:32.733 INFO:tasks.rook.operator.smithi057.stdout:   ... // 9 identical elements 2022-03-07T17:43:32.733 INFO:tasks.rook.operator.smithi057.stdout:   {Name: "smithi085-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi085-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-07T17:43:32.733 INFO:tasks.rook.operator.smithi057.stdout:   {Name: "smithi085-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi085-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-07T17:43:32.733 INFO:tasks.rook.operator.smithi057.stdout:+  { 2022-03-07T17:43:32.734 INFO:tasks.rook.operator.smithi057.stdout:+  Name: "smithi057-lv-1", 2022-03-07T17:43:32.734 INFO:tasks.rook.operator.smithi057.stdout:+  Count: 1, 2022-03-07T17:43:32.734 INFO:tasks.rook.operator.smithi057.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-07T17:43:32.734 INFO:tasks.rook.operator.smithi057.stdout:+  }, 2022-03-07T17:43:32.735 INFO:tasks.rook.operator.smithi057.stdout:   }, 2022-03-07T17:43:32.735 INFO:tasks.rook.operator.smithi057.stdout:   }, 2022-03-07T17:43:32.735 INFO:tasks.rook.operator.smithi057.stdout:   Annotations: nil, 2022-03-07T17:43:32.735 INFO:tasks.rook.operator.smithi057.stdout:   Labels: nil, 2022-03-07T17:43:32.736 INFO:tasks.rook.operator.smithi057.stdout:   ... // 20 identical fields 2022-03-07T17:43:32.736 INFO:tasks.rook.operator.smithi057.stdout:  } 2022-03-07T17:43:32.748 INFO:teuthology.orchestra.run.smithi057.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-07T17:43:32.813 INFO:teuthology.orchestra.run.smithi057.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-07T17:43:32.825 INFO:teuthology.orchestra.run.smithi057.stderr:+ num_osd=0 2022-03-07T17:43:32.825 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' 0 '!=' 12 ']' 2022-03-07T17:43:32.825 INFO:teuthology.orchestra.run.smithi057.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T17:43:32.826 INFO:teuthology.orchestra.run.smithi057.stdout:waiting for osd to come back up 2022-03-07T17:43:32.827 INFO:teuthology.orchestra.run.smithi057.stderr:++ toolbox ceph osd stat 2022-03-07T17:43:32.827 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f3 -d ' ' 2022-03-07T17:43:32.827 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T17:43:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:43:33.448 INFO:teuthology.orchestra.run.smithi057.stderr:+ num_osd=11 2022-03-07T17:43:33.448 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 30s 2022-03-07T17:43:36.253 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.245374 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:43:36.253 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.245396 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:43:36.254 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.245406 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:43:36.260 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.253571 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:43:36.381 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.376968 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:36.382 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.377160 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:36.909 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:36.905140 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:43:37.189 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:37.183392 I | op-mon: start running mons 2022-03-07T17:43:37.242 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:37.237658 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:43:37.317 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:37.311888 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.108.168.68:6789","10.102.1.250:6789","10.103.120.199:6789"]}] data:c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:43:37.476 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:37.471693 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:37.476 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:37.471978 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:38.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.164159 I | op-mon: targeting the mon count 3 2022-03-07T17:43:38.185 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.181070 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-07T17:43:38.190 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.181447 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:43:38.565 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.560661 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:43:38.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.582312 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:43:38.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.814735 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-07T17:43:38.850 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:38.845596 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T17:43:40.871 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:40.866668 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-07T17:43:46.712 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:46.703275 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:43:46.712 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:46.703301 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:43:46.712 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:46.703311 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:43:46.720 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:46.710999 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:43:46.781 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:46.773394 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:46.781 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:46.773620 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:47.276 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:47.272043 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:43:47.422 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:47.416445 I | op-mon: start running mons 2022-03-07T17:43:47.452 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:47.447596 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:43:47.468 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:47.461642 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.120.199:6789","10.108.168.68:6789","10.102.1.250:6789"]}] data:c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:43:47.475 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:47.471013 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:47.475 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:47.471176 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:48.704 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:48.699482 I | op-mon: targeting the mon count 3 2022-03-07T17:43:48.710 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:48.706015 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:43:49.142 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:49.137762 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:43:49.142 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:49.137785 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:43:49.565 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:49.560664 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:43:49.565 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:49.560704 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:43:49.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:49.946705 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:43:49.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:49.946734 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:43:50.312 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:50.307750 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:43:50.312 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:50.307791 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:43:50.715 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:50.710416 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:43:50.715 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:50.710449 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:43:51.090 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:51.086297 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:43:51.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:51.086317 I | op-mon: checking for basic quorum with existing mons 2022-03-07T17:43:51.852 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:51.848116 I | op-mon: mon "c" endpoint is [v2:10.108.168.68:3300,v1:10.108.168.68:6789] 2022-03-07T17:43:53.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:53.459029 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:43:54.164 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.160354 I | op-mon: mon "b" endpoint is [v2:10.103.120.199:3300,v1:10.103.120.199:6789] 2022-03-07T17:43:54.292 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.287517 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.108.168.68:6789","10.102.1.250:6789","10.103.120.199:6789"]}] data:c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:43:54.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.527590 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:43:54.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.541232 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:43:54.562 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.558048 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T17:43:54.571 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.566723 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T17:43:54.571 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.566756 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:43:54.600 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:54.596025 I | op-mon: mons running: [c a b] 2022-03-07T17:43:55.263 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:55.258541 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:43:55.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:55.543572 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T17:43:55.606 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:55.601959 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T17:43:55.607 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:55.601985 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:43:56.421 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:56.417247 I | op-mon: mons running: [c a b] 2022-03-07T17:43:57.290 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:57.285719 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:43:57.521 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:57.516430 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T17:43:57.529 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:57.525335 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T17:43:57.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:57.525362 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:43:57.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:57.552421 I | op-mon: mons running: [c a b] 2022-03-07T17:43:59.124 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:59.119385 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:43:59.124 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:59.119416 I | op-mon: mons created: 3 2022-03-07T17:43:59.659 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:59.654858 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:43:59.943 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:43:59.938525 I | op-mon: mons running: [c a b] 2022-03-07T17:44:00.593 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:00.589100 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:44:00.594 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:00.589129 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T17:44:01.145 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:01.141151 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T17:44:01.762 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:01.753221 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T17:44:02.798 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:02.793516 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T17:44:03.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:03.254587 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T17:44:03.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:03.254609 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T17:44:03.449 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' 11 '!=' 12 ']' 2022-03-07T17:44:03.449 INFO:teuthology.orchestra.run.smithi057.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T17:44:03.450 INFO:teuthology.orchestra.run.smithi057.stdout:waiting for osd to come back up 2022-03-07T17:44:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:++ toolbox ceph osd stat 2022-03-07T17:44:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T17:44:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f3 -d ' ' 2022-03-07T17:44:03.833 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:03.829093 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T17:44:03.869 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:44:04.348 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.343461 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T17:44:04.348 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.343522 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T17:44:04.349 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.343534 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T17:44:04.419 INFO:teuthology.orchestra.run.smithi057.stderr:+ num_osd=11 2022-03-07T17:44:04.419 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 30s 2022-03-07T17:44:04.761 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.757207 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T17:44:04.803 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.799143 I | ceph-spec: controller will reconcile resource "cluster-peer-token-rook-ceph" based on patch: [redacted patch details due to potentially sensitive content] 2022-03-07T17:44:05.004 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.999128 I | op-mgr: start running mgr 2022-03-07T17:44:05.004 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:04.999171 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T17:44:05.550 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.545468 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T17:44:05.574 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.569718 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T17:44:05.625 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.620254 I | op-mgr: setting services to point to mgr "a" 2022-03-07T17:44:05.765 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.760508 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T17:44:05.766 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.760541 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T17:44:05.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.810548 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T17:44:05.820 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814709 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-2" 2022-03-07T17:44:05.820 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814745 I | op-osd: OSD PVC "smithi057-lv-2-data-0gkc97" already exists 2022-03-07T17:44:05.820 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814757 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-3" 2022-03-07T17:44:05.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814767 I | op-osd: OSD PVC "smithi057-lv-3-data-07k2rn" already exists 2022-03-07T17:44:05.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814776 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-4" 2022-03-07T17:44:05.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814787 I | op-osd: OSD PVC "smithi057-lv-4-data-0h9pmf" already exists 2022-03-07T17:44:05.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814822 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-1" 2022-03-07T17:44:05.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814836 I | op-osd: OSD PVC "smithi066-lv-1-data-0gmmxm" already exists 2022-03-07T17:44:05.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814846 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-2" 2022-03-07T17:44:05.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814856 I | op-osd: OSD PVC "smithi066-lv-2-data-0tbn4m" already exists 2022-03-07T17:44:05.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814867 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-3" 2022-03-07T17:44:05.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814879 I | op-osd: OSD PVC "smithi066-lv-3-data-0pc4td" already exists 2022-03-07T17:44:05.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814904 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-4" 2022-03-07T17:44:05.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814916 I | op-osd: OSD PVC "smithi066-lv-4-data-052zdw" already exists 2022-03-07T17:44:05.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814925 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-1" 2022-03-07T17:44:05.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814936 I | op-osd: OSD PVC "smithi085-lv-1-data-0vtx9m" already exists 2022-03-07T17:44:05.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814946 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-2" 2022-03-07T17:44:05.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814958 I | op-osd: OSD PVC "smithi085-lv-2-data-08x59z" already exists 2022-03-07T17:44:05.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.814969 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-3" 2022-03-07T17:44:05.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.815002 I | op-osd: OSD PVC "smithi085-lv-3-data-0mzbnp" already exists 2022-03-07T17:44:05.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.815048 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-4" 2022-03-07T17:44:05.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.815082 I | op-osd: OSD PVC "smithi085-lv-4-data-0dkcqt" already exists 2022-03-07T17:44:05.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.815114 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-1" 2022-03-07T17:44:05.829 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.823203 I | op-osd: successfully provisioned PVC "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:05.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851391 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0gkc97" 2022-03-07T17:44:05.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851421 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-07k2rn" 2022-03-07T17:44:05.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851428 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:44:05.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851433 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:44:05.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851438 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:44:05.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851445 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0pc4td" 2022-03-07T17:44:05.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851451 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-052zdw" 2022-03-07T17:44:05.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851456 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:44:05.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851462 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-2-data-08x59z" 2022-03-07T17:44:05.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851467 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:44:05.859 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851472 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:44:05.859 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.851478 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:05.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.946183 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:05.952 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.946264 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T17:44:05.952 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:05.946308 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T17:44:06.075 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:06.070184 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-0mz4ml is "starting" 2022-03-07T17:44:07.543 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:07.538857 I | op-mgr: successful modules: prometheus 2022-03-07T17:44:07.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:07.541176 I | op-mgr: successful modules: dashboard 2022-03-07T17:44:07.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:07.550064 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T17:44:08.020 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:08.016224 I | op-osd: updating OSD 1 on PVC "smithi057-lv-3-data-07k2rn" 2022-03-07T17:44:08.021 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:08.016262 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-07k2rn" 2022-03-07T17:44:08.102 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:08.098065 I | op-osd: updating OSD 4 on PVC "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:44:08.103 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:08.098091 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:44:08.230 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:08.226056 I | op-mgr: successful modules: balancer 2022-03-07T17:44:10.124 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:10.119417 I | op-mgr: successful modules: orchestrator modules 2022-03-07T17:44:10.578 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:10.574135 I | op-osd: updating OSD 3 on PVC "smithi066-lv-3-data-0pc4td" 2022-03-07T17:44:10.579 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:10.574157 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0pc4td" 2022-03-07T17:44:11.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.021094 I | op-osd: updating OSD 5 on PVC "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:44:11.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.021224 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:44:11.341 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.337227 I | op-osd: updating OSD 6 on PVC "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:44:11.342 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.337266 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:44:11.668 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.663920 I | op-osd: updating OSD 7 on PVC "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:44:11.669 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.663951 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:44:11.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.936304 I | op-osd: updating OSD 8 on PVC "smithi085-lv-2-data-08x59z" 2022-03-07T17:44:11.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:11.936344 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-2-data-08x59z" 2022-03-07T17:44:12.144 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:12.139711 I | op-osd: updating OSD 9 on PVC "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:44:12.144 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:12.139736 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:44:12.312 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:12.307574 I | op-osd: updating OSD 10 on PVC "smithi066-lv-4-data-052zdw" 2022-03-07T17:44:12.312 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:12.307611 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-052zdw" 2022-03-07T17:44:12.446 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:12.441819 I | op-osd: updating OSD 11 on PVC "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:44:12.447 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:12.441843 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:44:15.300 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:15.294186 I | op-osd: updating OSD 2 on PVC "smithi057-lv-2-data-0gkc97" 2022-03-07T17:44:15.301 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:15.294212 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0gkc97" 2022-03-07T17:44:15.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:15.395312 I | op-osd: OSD orchestration status for node smithi057-lv-1-data-0mz4ml is "orchestrating" 2022-03-07T17:44:15.490 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:15.485414 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-0mz4ml is "orchestrating" 2022-03-07T17:44:21.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:21.408710 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-0mz4ml is "completed" 2022-03-07T17:44:21.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:21.408733 I | op-osd: creating OSD 0 on PVC "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:21.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:21.408741 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:22.996 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:22.991327 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T17:44:22.996 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:22.991355 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:44:23.008 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:23.002737 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:44:23.084 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:23.079492 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:44:23.107 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:23.103066 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2... 2022-03-07T17:44:29.990 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:29.985886 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10890 quincy" 2022-03-07T17:44:29.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:29.985908 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T17:44:29.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:29.985915 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10890 quincy", pursuing 2022-03-07T17:44:30.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:30.010790 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:44:30.018 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:30.013082 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:44:30.019 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:30.013587 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:44:30.534 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:30.529210 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10890 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2" 2022-03-07T17:44:31.160 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.153366 I | op-mon: start running mons 2022-03-07T17:44:31.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.306751 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:44:31.470 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.465575 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.108.168.68:6789","10.102.1.250:6789","10.103.120.199:6789"]}] data:c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:44:31.477 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.473390 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:44:31.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.473552 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:44:31.961 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.957164 I | op-mon: targeting the mon count 3 2022-03-07T17:44:31.971 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:31.966551 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:44:32.344 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:32.318679 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T17:44:32.344 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:32.318708 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:44:32.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:32.818898 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T17:44:32.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:32.818932 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:44:33.174 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:33.169674 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T17:44:33.175 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:33.169708 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:44:33.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:33.504096 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T17:44:33.509 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:33.504130 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:44:34.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:34.142301 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-07T17:44:34.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:34.142344 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T17:44:34.420 INFO:teuthology.orchestra.run.smithi057.stdout:waiting for osd to come back up 2022-03-07T17:44:34.421 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' 11 '!=' 12 ']' 2022-03-07T17:44:34.422 INFO:teuthology.orchestra.run.smithi057.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T17:44:34.422 INFO:teuthology.orchestra.run.smithi057.stderr:++ toolbox ceph osd stat 2022-03-07T17:44:34.422 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T17:44:34.422 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f3 -d ' ' 2022-03-07T17:44:34.747 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:44:34.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:34.906581 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T17:44:34.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:34.906606 I | op-mon: checking for basic quorum with existing mons 2022-03-07T17:44:35.449 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:35.444720 I | op-mon: mon "c" endpoint is [v2:10.108.168.68:3300,v1:10.108.168.68:6789] 2022-03-07T17:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:+ num_osd=11 2022-03-07T17:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 30s 2022-03-07T17:44:35.590 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:35.585842 I | op-mon: mon "a" endpoint is [v2:10.102.1.250:3300,v1:10.102.1.250:6789] 2022-03-07T17:44:35.734 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:35.729801 I | op-mon: mon "b" endpoint is [v2:10.103.120.199:3300,v1:10.103.120.199:6789] 2022-03-07T17:44:35.968 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:35.963473 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.1.250:6789","10.103.120.199:6789","10.108.168.68:6789"]}] data:c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 mapping:{"node":{"a":{"Name":"smithi085","Hostname":"smithi085","Address":"172.21.15.85"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"}}} maxMonId:2] 2022-03-07T17:44:36.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:36.562351 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T17:44:36.569 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:36.562669 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T17:44:36.992 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:36.987773 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T17:44:37.061 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:37.057000 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T17:44:37.062 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:37.057030 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:44:37.813 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:37.809031 I | op-mon: mons running: [c a b] 2022-03-07T17:44:38.236 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.232061 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:44:38.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.236934 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T17:44:38.248 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.244310 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T17:44:38.249 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.244332 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:44:38.273 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.268553 I | op-mon: mons running: [c a b] 2022-03-07T17:44:38.670 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.665094 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:44:38.679 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.675400 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T17:44:38.685 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.681420 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T17:44:38.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.681439 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:44:38.773 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:38.769278 I | op-mon: mons running: [c a b] 2022-03-07T17:44:39.190 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:39.185624 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:44:39.190 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:39.185648 I | op-mon: mons created: 3 2022-03-07T17:44:39.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:39.649228 I | op-mon: waiting for mon quorum with [c a b] 2022-03-07T17:44:39.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:39.681758 I | op-mon: mons running: [c a b] 2022-03-07T17:44:40.096 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:40.091662 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T17:44:40.096 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:40.091709 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T17:44:40.585 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:40.580706 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T17:44:41.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:41.011982 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T17:44:41.523 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:41.515672 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T17:44:41.970 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:41.965735 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T17:44:41.971 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:41.965765 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T17:44:42.482 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:42.478106 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T17:44:42.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:42.933019 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T17:44:42.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:42.933054 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T17:44:42.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:42.933062 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T17:44:43.360 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:43.356039 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T17:44:43.393 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:43.388405 I | op-mgr: start running mgr 2022-03-07T17:44:43.393 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:43.388437 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T17:44:43.885 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:43.880382 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T17:44:43.891 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:43.886903 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T17:44:43.894 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:43.889208 I | op-mgr: setting services to point to mgr "a" 2022-03-07T17:44:44.020 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.015792 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T17:44:44.021 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.015822 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T17:44:44.088 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.082214 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T17:44:44.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091886 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-2" 2022-03-07T17:44:44.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091926 I | op-osd: OSD PVC "smithi057-lv-2-data-0gkc97" already exists 2022-03-07T17:44:44.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091940 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-3" 2022-03-07T17:44:44.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091952 I | op-osd: OSD PVC "smithi057-lv-3-data-07k2rn" already exists 2022-03-07T17:44:44.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091967 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-4" 2022-03-07T17:44:44.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091979 I | op-osd: OSD PVC "smithi057-lv-4-data-0h9pmf" already exists 2022-03-07T17:44:44.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.091989 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-1" 2022-03-07T17:44:44.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092028 I | op-osd: OSD PVC "smithi066-lv-1-data-0gmmxm" already exists 2022-03-07T17:44:44.100 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092041 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-2" 2022-03-07T17:44:44.100 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092120 I | op-osd: OSD PVC "smithi066-lv-2-data-0tbn4m" already exists 2022-03-07T17:44:44.100 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092135 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-3" 2022-03-07T17:44:44.100 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092149 I | op-osd: OSD PVC "smithi066-lv-3-data-0pc4td" already exists 2022-03-07T17:44:44.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092161 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-4" 2022-03-07T17:44:44.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092172 I | op-osd: OSD PVC "smithi066-lv-4-data-052zdw" already exists 2022-03-07T17:44:44.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092182 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-1" 2022-03-07T17:44:44.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092192 I | op-osd: OSD PVC "smithi085-lv-1-data-0vtx9m" already exists 2022-03-07T17:44:44.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092202 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-2" 2022-03-07T17:44:44.102 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092213 I | op-osd: OSD PVC "smithi085-lv-2-data-08x59z" already exists 2022-03-07T17:44:44.102 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092226 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-3" 2022-03-07T17:44:44.102 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092237 I | op-osd: OSD PVC "smithi085-lv-3-data-0mzbnp" already exists 2022-03-07T17:44:44.103 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092245 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi085-lv-4" 2022-03-07T17:44:44.104 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092254 I | op-osd: OSD PVC "smithi085-lv-4-data-0dkcqt" already exists 2022-03-07T17:44:44.104 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092262 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-1" 2022-03-07T17:44:44.104 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.092271 I | op-osd: OSD PVC "smithi057-lv-1-data-0mz4ml" already exists 2022-03-07T17:44:44.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.125967 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0gkc97" 2022-03-07T17:44:44.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.125995 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-07k2rn" 2022-03-07T17:44:44.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126002 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:44:44.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126007 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:44:44.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126012 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:44:44.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126017 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0pc4td" 2022-03-07T17:44:44.133 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126022 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-052zdw" 2022-03-07T17:44:44.133 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126027 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:44:44.133 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126033 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-2-data-08x59z" 2022-03-07T17:44:44.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126038 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:44:44.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126043 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:44:44.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126049 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:44.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126054 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T17:44:44.135 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:44.126058 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T17:44:45.070 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.066009 I | op-mgr: successful modules: prometheus 2022-03-07T17:44:45.103 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.098749 I | op-mgr: successful modules: dashboard 2022-03-07T17:44:45.108 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.103398 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T17:44:45.332 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.327315 I | op-mgr: successful modules: balancer 2022-03-07T17:44:45.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.810204 I | op-osd: updating OSD 0 on PVC "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:45.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.810234 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-0mz4ml" 2022-03-07T17:44:45.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.853085 I | op-osd: updating OSD 1 on PVC "smithi057-lv-3-data-07k2rn" 2022-03-07T17:44:45.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.853111 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-07k2rn" 2022-03-07T17:44:45.945 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.941178 I | op-osd: updating OSD 4 on PVC "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:44:45.946 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:45.941203 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0h9pmf" 2022-03-07T17:44:46.748 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:46.743178 I | op-mgr: successful modules: orchestrator modules 2022-03-07T17:44:47.755 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.747585 I | op-osd: updating OSD 3 on PVC "smithi066-lv-3-data-0pc4td" 2022-03-07T17:44:47.755 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.747623 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0pc4td" 2022-03-07T17:44:47.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.833985 I | op-osd: updating OSD 5 on PVC "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:44:47.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.834017 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-0tbn4m" 2022-03-07T17:44:47.904 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.899144 I | op-osd: updating OSD 6 on PVC "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:44:47.904 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.899177 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-3-data-0mzbnp" 2022-03-07T17:44:47.972 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.967038 I | op-osd: updating OSD 7 on PVC "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:44:47.972 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:47.967079 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-1-data-0vtx9m" 2022-03-07T17:44:48.011 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.006985 I | op-osd: updating OSD 8 on PVC "smithi085-lv-2-data-08x59z" 2022-03-07T17:44:48.012 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.007013 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-2-data-08x59z" 2022-03-07T17:44:48.054 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.049468 I | op-osd: updating OSD 9 on PVC "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:44:48.054 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.049494 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0gmmxm" 2022-03-07T17:44:48.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.410704 I | op-osd: updating OSD 10 on PVC "smithi066-lv-4-data-052zdw" 2022-03-07T17:44:48.416 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.410746 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-052zdw" 2022-03-07T17:44:48.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.811047 I | op-osd: updating OSD 11 on PVC "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:44:48.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:48.811094 I | op-osd: OSD will have its main bluestore block on "smithi085-lv-4-data-0dkcqt" 2022-03-07T17:44:52.315 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:52.310858 I | op-osd: updating OSD 2 on PVC "smithi057-lv-2-data-0gkc97" 2022-03-07T17:44:52.316 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:52.310876 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0gkc97" 2022-03-07T17:44:52.795 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:52.790431 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T17:44:52.796 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:44:52.790458 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T17:45:05.496 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' 11 '!=' 12 ']' 2022-03-07T17:45:05.497 INFO:teuthology.orchestra.run.smithi057.stdout:waiting for osd to come back up 2022-03-07T17:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T17:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:++ toolbox ceph osd stat 2022-03-07T17:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T17:45:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:++ cut -f3 -d ' ' 2022-03-07T17:45:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T17:45:06.111 INFO:teuthology.orchestra.run.smithi057.stderr:+ num_osd=12 2022-03-07T17:45:06.111 INFO:teuthology.orchestra.run.smithi057.stderr:+ sleep 30s 2022-03-07T17:45:36.112 INFO:teuthology.orchestra.run.smithi057.stderr:+ '[' 12 '!=' 12 ']' 2022-03-07T17:45:36.114 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-07T17:45:36.125 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch status 2022-03-07T17:45:36.873 INFO:teuthology.orchestra.run.smithi057.stdout:Backend: rook 2022-03-07T17:45:36.874 INFO:teuthology.orchestra.run.smithi057.stdout:Available: Yes 2022-03-07T17:45:36.901 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ps 2022-03-07T17:45:37.463 INFO:teuthology.orchestra.run.smithi057.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-07T17:45:37.463 INFO:teuthology.orchestra.run.smithi057.stdout:crashcollector.crash smithi057 running (4m) 0s ago 4m - - 02e0d18029b5 2022-03-07T17:45:37.463 INFO:teuthology.orchestra.run.smithi057.stdout:crashcollector.crash smithi066 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.464 INFO:teuthology.orchestra.run.smithi057.stdout:crashcollector.crash smithi085 running (5m) 0s ago 5m - - 02e0d18029b5 2022-03-07T17:45:37.464 INFO:teuthology.orchestra.run.smithi057.stdout:mgr.a smithi066 running (5m) 0s ago 5m - - 02e0d18029b5 2022-03-07T17:45:37.465 INFO:teuthology.orchestra.run.smithi057.stdout:mon.a smithi085 running (7m) 0s ago 7m - - 02e0d18029b5 2022-03-07T17:45:37.465 INFO:teuthology.orchestra.run.smithi057.stdout:mon.b smithi057 running (7m) 0s ago 7m - - 02e0d18029b5 2022-03-07T17:45:37.465 INFO:teuthology.orchestra.run.smithi057.stdout:mon.c smithi066 running (6m) 0s ago 6m - - 02e0d18029b5 2022-03-07T17:45:37.465 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 smithi057 running (75s) 0s ago 75s - - 02e0d18029b5 2022-03-07T17:45:37.466 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 smithi057 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.466 INFO:teuthology.orchestra.run.smithi057.stdout:osd.10 smithi066 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.466 INFO:teuthology.orchestra.run.smithi057.stdout:osd.11 smithi085 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.466 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 smithi057 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.466 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 smithi066 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.467 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 smithi057 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.467 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 smithi066 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.467 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 smithi085 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.467 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 smithi085 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.468 INFO:teuthology.orchestra.run.smithi057.stdout:osd.8 smithi085 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.468 INFO:teuthology.orchestra.run.smithi057.stdout:osd.9 smithi066 running (3m) 0s ago 3m - - 02e0d18029b5 2022-03-07T17:45:37.482 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls 2022-03-07T17:45:38.175 INFO:teuthology.orchestra.run.smithi057.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-07T17:45:38.176 INFO:teuthology.orchestra.run.smithi057.stdout:crash 3/3 0s ago 5m * 2022-03-07T17:45:38.176 INFO:teuthology.orchestra.run.smithi057.stdout:mgr 1/1 0s ago 5m count:1 2022-03-07T17:45:38.176 INFO:teuthology.orchestra.run.smithi057.stdout:mon 3/3 0s ago 7m count:3 2022-03-07T17:45:38.176 INFO:teuthology.orchestra.run.smithi057.stdout:osd 24 0s ago 3m 2022-03-07T17:45:38.177 INFO:teuthology.orchestra.run.smithi057.stdout:osd.all-available-devices 0 0s ago - * 2022-03-07T17:45:38.201 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch host ls 2022-03-07T17:45:38.753 INFO:teuthology.orchestra.run.smithi057.stdout:HOST ADDR LABELS STATUS 2022-03-07T17:45:38.754 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 172.21.15.57/smithi057 2022-03-07T17:45:38.754 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 172.21.15.66/smithi066 2022-03-07T17:45:38.754 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 172.21.15.85/smithi085 2022-03-07T17:45:38.754 INFO:teuthology.orchestra.run.smithi057.stdout:3 hosts in cluster 2022-03-07T17:45:38.776 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch device ls 2022-03-07T17:45:39.349 INFO:teuthology.orchestra.run.smithi057.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-07T17:45:39.350 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T17:45:39.350 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T17:45:39.350 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T17:45:39.351 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T17:45:39.351 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T17:45:39.351 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T17:45:39.351 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T17:45:39.352 INFO:teuthology.orchestra.run.smithi057.stdout:smithi066 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T17:45:39.352 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T17:45:39.353 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T17:45:39.353 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T17:45:39.354 INFO:teuthology.orchestra.run.smithi057.stdout:smithi085 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T17:45:39.374 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch apply rgw foo 2022-03-07T17:45:39.931 INFO:teuthology.orchestra.run.smithi057.stdout:Created 2022-03-07T17:45:39.932 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:39.926417 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-07T17:45:39.951 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch apply mds foo 2022-03-07T17:45:40.040 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:40.035549 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:40.494 INFO:teuthology.orchestra.run.smithi057.stdout:Created 2022-03-07T17:45:40.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:40.489645 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-07T17:45:40.516 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch apply rbd-mirror 2022-03-07T17:45:40.553 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:40.548359 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:45:40.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:40.552930 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:41.037 INFO:teuthology.orchestra.run.smithi057.stdout:Success 2022-03-07T17:45:41.059 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch apply nfs foo --port 12777 2022-03-07T17:45:41.083 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.077968 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:41.227 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.221955 I | ceph-object-controller: reconciling object store deployments 2022-03-07T17:45:41.322 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.314603 I | ceph-object-controller: ceph object store gateway service running at 10.105.96.8 2022-03-07T17:45:41.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.314630 I | ceph-object-controller: reconciling object store pools 2022-03-07T17:45:41.575 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.570677 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-07T17:45:41.576 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.570714 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-07T17:45:41.576 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.570728 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-07T17:45:41.681 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:41.676533 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-07T17:45:42.697 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:42.690258 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-07T17:45:43.842 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:43.837361 I | ceph-file-controller: creating filesystem "foo" 2022-03-07T17:45:44.413 INFO:teuthology.orchestra.run.smithi057.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-07T17:45:44.413 INFO:teuthology.orchestra.run.smithi057.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-07T17:45:44.449 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-07T17:45:44.463 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:45:45.324 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:45:45.325 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:25.000000Z", "last_refresh": "2022-03-07T17:45:45.110219Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:45:45.110219Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:45:45.110219Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:45:45.110219Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:45:45.110219Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:45:45.110219Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-07T17:45:45.110219Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:45:45.110219Z", "running": 0, "size": 1}}] 2022-03-07T17:45:45.348 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch rm mds.foo 2022-03-07T17:45:45.989 INFO:teuthology.orchestra.run.smithi057.stdout:Removed cephfilesystems/foo 2022-03-07T17:45:46.010 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch rm rbd-mirror 2022-03-07T17:45:46.543 INFO:teuthology.orchestra.run.smithi057.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-07T17:45:46.567 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch rm rgw.foo 2022-03-07T17:45:46.591 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:46.585902 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-07T17:45:47.096 INFO:teuthology.orchestra.run.smithi057.stdout:Removed cephobjectstores/foo 2022-03-07T17:45:47.119 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:45:47.863 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:45:47.863 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:25.000000Z", "last_refresh": "2022-03-07T17:45:47.703139Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:45:47.703139Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:45:47.703139Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:45:47.703139Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:45:47.703139Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:45:47.703139Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:45:47.703139Z", "running": 0, "size": 1}}] 2022-03-07T17:45:49.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:49.254438 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-07T17:45:49.297 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:49.289018 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-07T17:45:49.309 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:49.291524 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-07T17:45:49.310 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:49.297200 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-07T17:45:49.310 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:49.303892 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-07T17:45:51.441 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:51.435994 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-07T17:45:51.976 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:51.964117 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:45:52.247 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:52.240792 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-07T17:45:53.528 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.523025 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-07T17:45:53.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.523781 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-07T17:45:53.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.523896 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-07T17:45:53.531 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.523976 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-07T17:45:53.531 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.524283 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-07T17:45:53.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.581817 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-07T17:45:53.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.581848 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-07T17:45:53.598 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:53.593653 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:56.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:56.334195 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-07T17:45:57.168 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.162113 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:57.151+0000 7f7628c6c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:57.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.151+0000 7f7628c6c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:57.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.151+0000 7f7628c6c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:57.170 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:57.170 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.162146 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:57.151+0000 7f7628c6c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:57.171 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.151+0000 7f7628c6c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:57.171 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.151+0000 7f7628c6c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:57.171 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:57.203 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.197595 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:57.627 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.621948 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-07T17:45:57.627 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.621999 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-07T17:45:57.799 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.793665 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:57.783+0000 7f4250644340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:57.799 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.783+0000 7f4250644340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:57.800 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.783+0000 7f4250644340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:57.800 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:57.800 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.793692 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:57.783+0000 7f4250644340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:57.800 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.783+0000 7f4250644340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:57.801 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:57.783+0000 7f4250644340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:57.801 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:57.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:57.809873 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:57.893 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:45:58.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:58.407857 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:58.399+0000 7f1042786340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:58.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:58.399+0000 7f1042786340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:58.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:58.399+0000 7f1042786340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:58.414 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:58.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:58.407879 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:58.399+0000 7f1042786340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:58.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:58.399+0000 7f1042786340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:58.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:58.399+0000 7f1042786340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:58.415 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:58.423 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:58.418217 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:58.633 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:45:58.634 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:25.000000Z", "last_refresh": "2022-03-07T17:45:58.432706Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:45:58.432706Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:45:58.432706Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:45:58.432706Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:45:58.432706Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:45:58.432706Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:45:58.432706Z", "running": 0, "size": 1}}] 2022-03-07T17:45:58.862 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:58.857112 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-07T17:45:59.018 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.012519 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:59.007+0000 7f873b030340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:59.018 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.007+0000 7f873b030340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:59.018 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.007+0000 7f873b030340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:59.019 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:59.019 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.012549 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:59.007+0000 7f873b030340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:59.019 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.007+0000 7f873b030340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:59.019 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.007+0000 7f873b030340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:59.020 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:59.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.026476 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:45:59.275 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.269860 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-07T17:45:59.603 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.597959 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:59.591+0000 7f2969567340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:59.603 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.591+0000 7f2969567340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:59.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.591+0000 7f2969567340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:59.604 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:59.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.597991 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:45:59.591+0000 7f2969567340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:45:59.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.591+0000 7f2969567340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:45:59.605 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:45:59.591+0000 7f2969567340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:45:59.605 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:45:59.703 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:45:59.698215 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:00.748 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:00.743510 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-07T17:46:00.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:00.810869 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:00.803+0000 7effe68bc340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:00.816 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:00.803+0000 7effe68bc340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:00.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:00.803+0000 7effe68bc340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:00.817 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:00.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:00.810896 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:00.803+0000 7effe68bc340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:00.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:00.803+0000 7effe68bc340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:00.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:00.803+0000 7effe68bc340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:00.818 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:00.831 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:00.826376 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:01.165 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:01.160503 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-07T17:46:01.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:01.405429 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:01.399+0000 7f023bca5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:01.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:01.399+0000 7f023bca5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:01.412 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:01.399+0000 7f023bca5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:01.412 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:01.412 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:01.405455 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:01.399+0000 7f023bca5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:01.412 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:01.399+0000 7f023bca5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:01.412 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:01.399+0000 7f023bca5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:01.413 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:01.431 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:01.426352 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:01.708 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:01.703778 I | op-mds: setting mds config flags 2022-03-07T17:46:01.709 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:01.703800 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T17:46:02.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:02.111790 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:02.103+0000 7f3dfbf81340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:02.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:02.103+0000 7f3dfbf81340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:02.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:02.103+0000 7f3dfbf81340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:02.118 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:02.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:02.111815 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:02.103+0000 7f3dfbf81340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:02.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:02.103+0000 7f3dfbf81340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:02.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:02.103+0000 7f3dfbf81340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:02.119 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:02.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:02.144211 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:02.170 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:02.165086 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T17:46:02.177 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:02.172647 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-07T17:46:02.632 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:02.618218 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:46:03.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:03.229270 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:03.223+0000 7ff03cf0b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:03.236 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:03.223+0000 7ff03cf0b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:03.236 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:03.223+0000 7ff03cf0b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:03.237 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:03.237 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:03.229304 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:03.223+0000 7ff03cf0b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:03.238 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:03.223+0000 7ff03cf0b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:03.238 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:03.223+0000 7ff03cf0b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:03.239 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:03.403 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:03.398003 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:03.657 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:03.652090 I | op-mds: setting mds config flags 2022-03-07T17:46:03.657 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:03.652116 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T17:46:04.368 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:04.362724 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:04.355+0000 7f7f83121340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:04.368 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:04.355+0000 7f7f83121340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:04.369 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:04.355+0000 7f7f83121340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:04.369 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:04.369 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:04.362751 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:04.355+0000 7f7f83121340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:04.369 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:04.355+0000 7f7f83121340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:04.370 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:04.355+0000 7f7f83121340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:04.370 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:04.707 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:04.701750 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T17:46:04.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:04.922821 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:06.448 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:06.443363 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-07T17:46:07.829 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:07.823906 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-07T17:46:07.830 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:07.823944 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-07T17:46:07.830 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:07.823958 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-07T17:46:08.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:08.551324 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-07T17:46:08.657 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:46:09.400 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:46:09.400 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:25.000000Z", "last_refresh": "2022-03-07T17:46:09.187107Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:09.187107Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:46:09.187107Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:46:09.187107Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:46:09.187107Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:46:09.187107Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:09.187107Z", "running": 0, "size": 1}}] 2022-03-07T17:46:09.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:09.489946 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:09.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:09.763669 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-07T17:46:10.409 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:10.403686 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:10.395+0000 7f0830b4a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:10.409 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:10.395+0000 7f0830b4a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:10.410 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:10.395+0000 7f0830b4a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:10.410 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:10.410 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:10.403715 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:10.395+0000 7f0830b4a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:10.410 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:10.395+0000 7f0830b4a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:10.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:10.395+0000 7f0830b4a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:10.411 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:10.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:10.524915 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-07T17:46:10.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:10.524943 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-07T17:46:10.531 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:10.524949 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-07T17:46:11.440 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:11.435394 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-07T17:46:11.441 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:11.435426 I | ceph-file-controller: downing filesystem "foo" 2022-03-07T17:46:13.233 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:13.226293 I | ceph-file-controller: downed filesystem "foo" 2022-03-07T17:46:16.445 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:16.438588 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-07T17:46:16.446 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:16.438610 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-07T17:46:19.429 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:46:20.364 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:46:20.364 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:25.000000Z", "last_refresh": "2022-03-07T17:46:20.161317Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:20.161317Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:46:20.161317Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:46:20.161317Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:46:20.161317Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:46:20.161317Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:20.161317Z", "running": 0, "size": 1}}] 2022-03-07T17:46:20.666 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:20.661013 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:20.679 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:20.674279 I | cephclient: purge completed for pool "foo-metadata" 2022-03-07T17:46:21.103 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:21.098260 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-07T17:46:21.103 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:21.098286 I | cephclient: purging pool "foo-data0" (id=9) 2022-03-07T17:46:21.292 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:21.286981 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:21.278+0000 7f482401f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:21.293 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:21.278+0000 7f482401f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:21.293 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:21.278+0000 7f482401f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:21.293 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:21.293 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:21.287009 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:21.278+0000 7f482401f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:21.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:21.278+0000 7f482401f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:21.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:21.278+0000 7f482401f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:21.294 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:23.473 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:23.467829 I | cephclient: purge completed for pool "foo-data0" 2022-03-07T17:46:23.474 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:23.467882 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-07T17:46:23.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:23.473469 E | ceph-file-controller: failed to reconcile failed to remove finalizer: failed to remove finalizer "cephfilesystem.ceph.rook.io" on "foo": Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:46:23.488 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:23.483203 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:24.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:24.314051 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-07T17:46:24.361 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:24.355938 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-07T17:46:24.397 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:24.391414 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-07T17:46:24.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:24.392221 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-07T17:46:24.400 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:24.392736 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-07T17:46:27.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:27.393840 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:46:27.552 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:27.546862 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi085": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T17:46:30.387 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:46:34.509 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:46:34.510 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:46:34.265122Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:46:34.265122Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:46:34.265122Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:46:34.265122Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:46:34.265122Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:34.265122Z", "running": 0, "size": 1}}] 2022-03-07T17:46:41.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:41.771668 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:46:42.435 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:42.422750 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:42.414+0000 7fea5a83c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:42.435 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:42.414+0000 7fea5a83c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:42.435 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:42.414+0000 7fea5a83c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:42.435 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:42.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:46:42.422779 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:46:42.414+0000 7fea5a83c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:46:42.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:42.414+0000 7fea5a83c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:46:42.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:46:42.414+0000 7fea5a83c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:46:42.436 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:46:44.535 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:46:45.254 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:46:45.254 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:46:45.068685Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:46:45.068685Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:46:45.068685Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:46:45.068685Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:46:45.068685Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:45.068685Z", "running": 0, "size": 1}}] 2022-03-07T17:46:55.280 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:46:55.985 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:46:55.985 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:46:55.799567Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:46:55.799567Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:46:55.799567Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:46:55.799567Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:46:55.799567Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:46:55.799567Z", "running": 0, "size": 1}}] 2022-03-07T17:47:06.017 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:47:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:47:06.805 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:47:06.553829Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:47:06.553829Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:47:06.553829Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:47:06.553829Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:47:06.553829Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:47:06.553829Z", "running": 0, "size": 1}}] 2022-03-07T17:47:16.837 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:47:17.556 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:47:17.557 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:47:17.374052Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:47:17.374052Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:47:17.374052Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:47:17.374052Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:47:17.374052Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:47:17.374052Z", "running": 0, "size": 1}}] 2022-03-07T17:47:23.405 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:47:23.398623 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:47:24.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:47:24.125213 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:47:24.113+0000 7f16d9ad3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:47:24.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:47:24.113+0000 7f16d9ad3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:47:24.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:47:24.113+0000 7f16d9ad3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:47:24.132 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:47:24.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:47:24.125309 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:47:24.113+0000 7f16d9ad3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:47:24.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:47:24.113+0000 7f16d9ad3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:47:24.133 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:47:24.113+0000 7f16d9ad3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:47:24.133 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:47:27.584 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:47:28.320 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:47:28.321 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:47:28.118328Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:47:28.118328Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:47:28.118328Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:47:28.118328Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:47:28.118328Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:47:28.118328Z", "running": 0, "size": 1}}] 2022-03-07T17:47:38.347 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:47:39.057 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:47:39.057 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:47:38.882841Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:47:38.882841Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:47:38.882841Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:47:38.882841Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:47:38.882841Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:47:38.882841Z", "running": 0, "size": 1}}] 2022-03-07T17:47:49.087 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:47:49.803 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:47:49.803 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:47:49.618455Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:47:49.618455Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:47:49.618455Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:47:49.618455Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:47:49.618455Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:47:49.618455Z", "running": 0, "size": 1}}] 2022-03-07T17:47:59.829 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:48:00.515 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:48:00.516 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:48:00.330006Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:48:00.330006Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:48:00.330006Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:48:00.330006Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:48:00.330006Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:48:00.330006Z", "running": 0, "size": 1}}] 2022-03-07T17:48:10.536 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:48:11.262 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:48:11.263 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:48:11.062590Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:48:11.062590Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:48:11.062590Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:48:11.062590Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:48:11.062590Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:48:11.062590Z", "running": 0, "size": 1}}] 2022-03-07T17:48:21.289 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:48:22.199 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:48:22.199 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:48:21.957166Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:48:21.957166Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:48:21.957166Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:48:21.957166Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:48:21.957166Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:48:21.957166Z", "running": 0, "size": 1}}] 2022-03-07T17:48:32.226 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:48:32.947 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:48:32.948 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:48:32.779406Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:48:32.779406Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:48:32.779406Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:48:32.779406Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:48:32.779406Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:48:32.779406Z", "running": 0, "size": 1}}] 2022-03-07T17:48:42.976 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:48:43.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:48:43.682 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:48:43.496876Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:48:43.496876Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:48:43.496876Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:48:43.496876Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:48:43.496876Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:48:43.496876Z", "running": 0, "size": 1}}] 2022-03-07T17:48:46.060 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:48:46.054429 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:48:46.709 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:48:46.702808 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:48:46.692+0000 7f95e9ac8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:48:46.709 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:48:46.692+0000 7f95e9ac8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:48:46.710 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:48:46.692+0000 7f95e9ac8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:48:46.710 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:48:46.710 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:48:46.702844 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:48:46.692+0000 7f95e9ac8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:48:46.710 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:48:46.692+0000 7f95e9ac8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:48:46.711 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:48:46.692+0000 7f95e9ac8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:48:46.711 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:48:53.706 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:48:54.443 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:48:54.443 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:48:54.266965Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:48:54.266965Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:48:54.266965Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:48:54.266965Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:48:54.266965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:48:54.266965Z", "running": 0, "size": 1}}] 2022-03-07T17:49:04.466 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:49:05.193 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:49:05.194 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:49:05.011486Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:49:05.011486Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:49:05.011486Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:49:05.011486Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:49:05.011486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:49:05.011486Z", "running": 0, "size": 1}}] 2022-03-07T17:49:15.219 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:49:16.243 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:49:16.243 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:49:15.748380Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:49:15.748380Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:49:15.748380Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:49:15.748380Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:49:15.748380Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:49:15.748380Z", "running": 0, "size": 1}}] 2022-03-07T17:49:26.270 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:49:27.001 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:49:27.002 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:49:26.818083Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:49:26.818083Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:49:26.818083Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:49:26.818083Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:49:26.818083Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:49:26.818083Z", "running": 0, "size": 1}}] 2022-03-07T17:49:37.030 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:49:37.747 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:49:37.748 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:49:37.544186Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:49:37.544186Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:49:37.544186Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:49:37.544186Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:49:37.544186Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:49:37.544186Z", "running": 0, "size": 1}}] 2022-03-07T17:49:47.774 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:49:48.587 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:49:48.587 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:49:48.414446Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:49:48.414446Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:49:48.414446Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:49:48.414446Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:49:48.414446Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:49:48.414446Z", "running": 0, "size": 1}}] 2022-03-07T17:49:58.612 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:49:59.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:49:59.328 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:49:59.132505Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:49:59.132505Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:49:59.132505Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:49:59.132505Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:49:59.132505Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:49:59.132505Z", "running": 0, "size": 1}}] 2022-03-07T17:50:09.354 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:50:10.121 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:50:10.122 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:50:09.948927Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:50:09.948927Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:50:09.948927Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:50:09.948927Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:50:09.948927Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:50:09.948927Z", "running": 0, "size": 1}}] 2022-03-07T17:50:20.148 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:50:20.951 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:50:20.951 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:50:20.720748Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:50:20.720748Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:50:20.720748Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:50:20.720748Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:50:20.720748Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:50:20.720748Z", "running": 0, "size": 1}}] 2022-03-07T17:50:30.978 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:50:31.651 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:50:31.687 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:50:31.477128Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:50:31.477128Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:50:31.477128Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:50:31.477128Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:50:31.477128Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:50:31.477128Z", "running": 0, "size": 1}}] 2022-03-07T17:50:41.689 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:50:42.463 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:50:42.463 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:50:42.236191Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:50:42.236191Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:50:42.236191Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:50:42.236191Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:50:42.236191Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:50:42.236191Z", "running": 0, "size": 1}}] 2022-03-07T17:50:52.491 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:50:53.178 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:50:53.178 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:50:52.975002Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:50:52.975002Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:50:52.975002Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:50:52.975002Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:50:52.975002Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:50:52.975002Z", "running": 0, "size": 1}}] 2022-03-07T17:51:03.202 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:51:03.995 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:51:03.996 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:51:03.798027Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:51:03.798027Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:51:03.798027Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:51:03.798027Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:51:03.798027Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:51:03.798027Z", "running": 0, "size": 1}}] 2022-03-07T17:51:14.017 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:51:14.738 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:51:14.738 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:51:14.542128Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:51:14.542128Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:51:14.542128Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:51:14.542128Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:51:14.542128Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:51:14.542128Z", "running": 0, "size": 1}}] 2022-03-07T17:51:24.764 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:51:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:51:25.475 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:51:25.307084Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:51:25.307084Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:51:25.307084Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:51:25.307084Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:51:25.307084Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:51:25.307084Z", "running": 0, "size": 1}}] 2022-03-07T17:51:30.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:51:30.552379 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:51:31.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:51:31.191531 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:51:31.181+0000 7fcd24090340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:51:31.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:51:31.181+0000 7fcd24090340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:51:31.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:51:31.181+0000 7fcd24090340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:51:31.199 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:51:31.201 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:51:31.191580 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:51:31.181+0000 7fcd24090340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:51:31.202 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:51:31.181+0000 7fcd24090340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:51:31.202 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:51:31.181+0000 7fcd24090340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:51:31.202 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:51:35.502 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:51:36.195 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:51:36.195 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:51:36.025656Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:51:36.025656Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:51:36.025656Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:51:36.025656Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:51:36.025656Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:51:36.025656Z", "running": 0, "size": 1}}] 2022-03-07T17:51:46.218 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:51:46.989 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:51:46.990 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:51:46.789107Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:51:46.789107Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:51:46.789107Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:51:46.789107Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:51:46.789107Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:51:46.789107Z", "running": 0, "size": 1}}] 2022-03-07T17:51:57.017 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:51:57.749 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:51:57.749 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:51:57.537674Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:51:57.537674Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:51:57.537674Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:51:57.537674Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:51:57.537674Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:51:57.537674Z", "running": 0, "size": 1}}] 2022-03-07T17:52:07.776 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:52:08.490 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:52:08.490 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:52:08.305659Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:52:08.305659Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:52:08.305659Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:52:08.305659Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:52:08.305659Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:52:08.305659Z", "running": 0, "size": 1}}] 2022-03-07T17:52:18.523 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:52:19.228 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:52:19.228 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:52:19.054851Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:52:19.054851Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:52:19.054851Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:52:19.054851Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:52:19.054851Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:52:19.054851Z", "running": 0, "size": 1}}] 2022-03-07T17:52:29.253 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:52:29.979 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:52:29.980 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:52:29.794459Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:52:29.794459Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:52:29.794459Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:52:29.794459Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:52:29.794459Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:52:29.794459Z", "running": 0, "size": 1}}] 2022-03-07T17:52:40.004 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:52:40.829 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:52:40.829 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:52:40.545939Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:52:40.545939Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:52:40.545939Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:52:40.545939Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:52:40.545939Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:52:40.545939Z", "running": 0, "size": 1}}] 2022-03-07T17:52:50.859 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:52:51.555 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:52:51.556 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:52:51.374535Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:52:51.374535Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:52:51.374535Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:52:51.374535Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:52:51.374535Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:52:51.374535Z", "running": 0, "size": 1}}] 2022-03-07T17:53:01.582 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:53:02.290 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:53:02.291 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:53:02.092020Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:53:02.092020Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:53:02.092020Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:53:02.092020Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:53:02.092020Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:53:02.092020Z", "running": 0, "size": 1}}] 2022-03-07T17:53:12.317 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:53:13.041 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:53:13.042 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:53:12.847423Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:53:12.847423Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:53:12.847423Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:53:12.847423Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:53:12.847423Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:53:12.847423Z", "running": 0, "size": 1}}] 2022-03-07T17:53:23.064 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:53:23.834 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:53:23.834 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:53:23.650404Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:53:23.650404Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:53:23.650404Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:53:23.650404Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:53:23.650404Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:53:23.650404Z", "running": 0, "size": 1}}] 2022-03-07T17:53:33.860 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:53:34.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:53:34.664 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:53:34.387956Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:53:34.387956Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:53:34.387956Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:53:34.387956Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:53:34.387956Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:53:34.387956Z", "running": 0, "size": 1}}] 2022-03-07T17:53:44.687 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:53:45.458 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:53:45.458 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:53:45.255141Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:53:45.255141Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:53:45.255141Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:53:45.255141Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:53:45.255141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:53:45.255141Z", "running": 0, "size": 1}}] 2022-03-07T17:53:55.485 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:53:56.205 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:53:56.205 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:53:56.014942Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:53:56.014942Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:53:56.014942Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:53:56.014942Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:53:56.014942Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:53:56.014942Z", "running": 0, "size": 1}}] 2022-03-07T17:54:06.229 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:54:06.939 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:54:06.939 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:54:06.751214Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:54:06.751214Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:54:06.751214Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:54:06.751214Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:54:06.751214Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:54:06.751214Z", "running": 0, "size": 1}}] 2022-03-07T17:54:16.969 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:54:17.758 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:54:17.758 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:54:17.564706Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:54:17.564706Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:54:17.564706Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:54:17.564706Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:54:17.564706Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:54:17.564706Z", "running": 0, "size": 1}}] 2022-03-07T17:54:27.783 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:54:28.845 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:54:28.845 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:54:28.326376Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:54:28.326376Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:54:28.326376Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:54:28.326376Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:54:28.326376Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:54:28.326376Z", "running": 0, "size": 1}}] 2022-03-07T17:54:38.876 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:54:39.590 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:54:39.590 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:54:39.419595Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:54:39.419595Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:54:39.419595Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:54:39.419595Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:54:39.419595Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:54:39.419595Z", "running": 0, "size": 1}}] 2022-03-07T17:54:49.639 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:54:50.334 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:54:50.334 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:54:50.151521Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:54:50.151521Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:54:50.151521Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:54:50.151521Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:54:50.151521Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:54:50.151521Z", "running": 0, "size": 1}}] 2022-03-07T17:55:00.358 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:55:01.103 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:55:01.104 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:55:00.893651Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:55:00.893651Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:55:00.893651Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:55:00.893651Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:55:00.893651Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:55:00.893651Z", "running": 0, "size": 1}}] 2022-03-07T17:55:11.131 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:55:11.826 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:55:11.826 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:55:11.653888Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:55:11.653888Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:55:11.653888Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:55:11.653888Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:55:11.653888Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:55:11.653888Z", "running": 0, "size": 1}}] 2022-03-07T17:55:21.852 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:55:22.585 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:55:22.586 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:55:22.393888Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:55:22.393888Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:55:22.393888Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:55:22.393888Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:55:22.393888Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:55:22.393888Z", "running": 0, "size": 1}}] 2022-03-07T17:55:32.614 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:55:33.394 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:55:33.395 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:55:33.125752Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:55:33.125752Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:55:33.125752Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:55:33.125752Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:55:33.125752Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:55:33.125752Z", "running": 0, "size": 1}}] 2022-03-07T17:55:43.422 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:55:44.207 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:55:44.207 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:55:43.962433Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:55:43.962433Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:55:43.962433Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:55:43.962433Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:55:43.962433Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:55:43.962433Z", "running": 0, "size": 1}}] 2022-03-07T17:55:54.227 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:55:54.985 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:55:54.986 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:55:54.767656Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:55:54.767656Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:55:54.767656Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:55:54.767656Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:55:54.767656Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:55:54.767656Z", "running": 0, "size": 1}}] 2022-03-07T17:56:05.012 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:56:05.740 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:56:05.740 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:56:05.545924Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:56:05.545924Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:56:05.545924Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:56:05.545924Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:56:05.545924Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:56:05.545924Z", "running": 0, "size": 1}}] 2022-03-07T17:56:15.769 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:56:16.496 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:56:16.496 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:56:16.309461Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:56:16.309461Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:56:16.309461Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:56:16.309461Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:56:16.309461Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:56:16.309461Z", "running": 0, "size": 1}}] 2022-03-07T17:56:26.524 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:56:27.231 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:56:27.232 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:56:27.051537Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:56:27.051537Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:56:27.051537Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:56:27.051537Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:56:27.051537Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:56:27.051537Z", "running": 0, "size": 1}}] 2022-03-07T17:56:37.258 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:56:40.982 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:56:40.982 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:56:40.803883Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:56:40.803883Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:56:40.803883Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:56:40.803883Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:56:40.803883Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:56:40.803883Z", "running": 0, "size": 1}}] 2022-03-07T17:56:51.007 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:56:52.030 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:56:52.031 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:56:51.830317Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:56:51.830317Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:56:51.830317Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:56:51.830317Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:56:51.830317Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:56:51.830317Z", "running": 0, "size": 1}}] 2022-03-07T17:56:58.910 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:56:58.903512 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T17:56:59.563 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:56:59.555153 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:56:59.544+0000 7fecd5110340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:56:59.563 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:56:59.544+0000 7fecd5110340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:56:59.563 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:56:59.544+0000 7fecd5110340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:56:59.563 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:56:59.564 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 17:56:59.555188 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T17:56:59.544+0000 7fecd5110340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T17:56:59.564 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:56:59.544+0000 7fecd5110340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T17:56:59.564 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T17:56:59.544+0000 7fecd5110340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T17:56:59.564 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T17:57:02.055 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:57:02.851 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:57:02.851 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:57:02.670004Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:57:02.670004Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:57:02.670004Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:57:02.670004Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:57:02.670004Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:57:02.670004Z", "running": 0, "size": 1}}] 2022-03-07T17:57:12.877 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:57:13.554 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:57:13.555 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:57:13.348397Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:57:13.348397Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:57:13.348397Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:57:13.348397Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:57:13.348397Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:57:13.348397Z", "running": 0, "size": 1}}] 2022-03-07T17:57:23.580 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:57:24.351 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:57:24.351 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:57:24.136804Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:57:24.136804Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:57:24.136804Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:57:24.136804Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:57:24.136804Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:57:24.136804Z", "running": 0, "size": 1}}] 2022-03-07T17:57:34.378 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:57:35.064 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:57:35.064 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:57:34.886188Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:57:34.886188Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:57:34.886188Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:57:34.886188Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:57:34.886188Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:57:34.886188Z", "running": 0, "size": 1}}] 2022-03-07T17:57:45.090 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:57:45.791 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:57:45.791 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:57:45.611996Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:57:45.611996Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:57:45.611996Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:57:45.611996Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:57:45.611996Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:57:45.611996Z", "running": 0, "size": 1}}] 2022-03-07T17:57:55.818 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:57:56.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:57:56.665 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:57:56.465876Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:57:56.465876Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:57:56.465876Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:57:56.465876Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:57:56.465876Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:57:56.465876Z", "running": 0, "size": 1}}] 2022-03-07T17:58:06.691 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:58:07.394 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:58:07.395 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:58:07.198345Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:58:07.198345Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:58:07.198345Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:58:07.198345Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:58:07.198345Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:58:07.198345Z", "running": 0, "size": 1}}] 2022-03-07T17:58:17.423 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:58:18.169 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:58:18.170 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:58:17.967913Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:58:17.967913Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:58:17.967913Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:58:17.967913Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:58:17.967913Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:58:17.967913Z", "running": 0, "size": 1}}] 2022-03-07T17:58:28.194 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:58:28.899 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:58:28.899 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:58:28.707892Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:58:28.707892Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:58:28.707892Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:58:28.707892Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:58:28.707892Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:58:28.707892Z", "running": 0, "size": 1}}] 2022-03-07T17:58:38.923 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:58:39.655 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:58:39.655 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:58:39.458412Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:58:39.458412Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:58:39.458412Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:58:39.458412Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:58:39.458412Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:58:39.458412Z", "running": 0, "size": 1}}] 2022-03-07T17:58:49.683 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:58:50.386 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:58:50.386 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:58:50.200261Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:58:50.200261Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:58:50.200261Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:58:50.200261Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:58:50.200261Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:58:50.200261Z", "running": 0, "size": 1}}] 2022-03-07T17:59:00.411 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:59:01.125 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:59:01.125 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:59:00.924975Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:59:00.924975Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:59:00.924975Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:59:00.924975Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:59:00.924975Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:59:00.924975Z", "running": 0, "size": 1}}] 2022-03-07T17:59:11.152 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:59:11.883 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:59:11.884 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:59:11.702578Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:59:11.702578Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:59:11.702578Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:59:11.702578Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:59:11.702578Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:59:11.702578Z", "running": 0, "size": 1}}] 2022-03-07T17:59:21.910 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:59:22.649 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:59:22.649 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:59:22.456306Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:59:22.456306Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:59:22.456306Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:59:22.456306Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:59:22.456306Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:59:22.456306Z", "running": 0, "size": 1}}] 2022-03-07T17:59:32.672 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:59:33.402 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:59:33.403 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:59:33.205290Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:59:33.205290Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:59:33.205290Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:59:33.205290Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:59:33.205290Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:59:33.205290Z", "running": 0, "size": 1}}] 2022-03-07T17:59:43.428 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:59:44.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:59:44.353 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:59:44.083660Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:59:44.083660Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:59:44.083660Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:59:44.083660Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:59:44.083660Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:59:44.083660Z", "running": 0, "size": 1}}] 2022-03-07T17:59:54.355 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T17:59:55.092 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T17:59:55.093 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T17:59:54.878425Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T17:59:54.878425Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T17:59:54.878425Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T17:59:54.878425Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T17:59:54.878425Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T17:59:54.878425Z", "running": 0, "size": 1}}] 2022-03-07T18:00:05.115 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:00:05.826 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:00:05.826 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:00:05.634471Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:00:05.634471Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:00:05.634471Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:00:05.634471Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:00:05.634471Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:00:05.634471Z", "running": 0, "size": 1}}] 2022-03-07T18:00:15.853 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:00:16.895 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:00:16.895 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:00:16.380721Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:00:16.380721Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:00:16.380721Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:00:16.380721Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:00:16.380721Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:00:16.380721Z", "running": 0, "size": 1}}] 2022-03-07T18:00:26.920 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:00:27.701 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:00:27.701 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:00:27.461169Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:00:27.461169Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:00:27.461169Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:00:27.461169Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:00:27.461169Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:00:27.461169Z", "running": 0, "size": 1}}] 2022-03-07T18:00:37.725 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:00:38.556 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:00:38.557 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:00:38.247381Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:00:38.247381Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:00:38.247381Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:00:38.247381Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:00:38.247381Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:00:38.247381Z", "running": 0, "size": 1}}] 2022-03-07T18:00:48.581 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:00:49.272 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:00:49.272 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:00:49.088122Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:00:49.088122Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:00:49.088122Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:00:49.088122Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:00:49.088122Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:00:49.088122Z", "running": 0, "size": 1}}] 2022-03-07T18:00:59.301 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:01:00.047 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:01:00.048 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:00:59.864549Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:00:59.864549Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:00:59.864549Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:00:59.864549Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:00:59.864549Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:00:59.864549Z", "running": 0, "size": 1}}] 2022-03-07T18:01:10.074 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:01:10.834 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:01:10.835 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:01:10.623890Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:01:10.623890Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:01:10.623890Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:01:10.623890Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:01:10.623890Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:01:10.623890Z", "running": 0, "size": 1}}] 2022-03-07T18:01:20.860 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:01:21.589 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:01:21.589 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:01:21.388360Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:01:21.388360Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:01:21.388360Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:01:21.388360Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:01:21.388360Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:01:21.388360Z", "running": 0, "size": 1}}] 2022-03-07T18:01:31.616 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:01:32.364 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:01:32.364 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:01:32.170420Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:01:32.170420Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:01:32.170420Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:01:32.170420Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:01:32.170420Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:01:32.170420Z", "running": 0, "size": 1}}] 2022-03-07T18:01:42.395 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:01:43.143 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:01:43.143 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:01:42.935131Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:01:42.935131Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:01:42.935131Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:01:42.935131Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:01:42.935131Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:01:42.935131Z", "running": 0, "size": 1}}] 2022-03-07T18:01:53.168 DEBUG:teuthology.orchestra.run.smithi057:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-mchzd -- ceph orch ls -f json 2022-03-07T18:01:53.871 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-07T18:01:53.871 INFO:teuthology.orchestra.run.smithi057.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:42.000000Z", "last_refresh": "2022-03-07T18:01:53.680391Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:40:16.000000Z", "last_refresh": "2022-03-07T18:01:53.680391Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:37:55.000000Z", "last_refresh": "2022-03-07T18:01:53.680391Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02e0d18029b5d43b9a966015607bd55744947092bb79180495ed57e73ea798da", "container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "created": "2022-03-07T17:41:59.000000Z", "last_refresh": "2022-03-07T18:01:53.680391Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T18:01:53.680391Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2", "last_refresh": "2022-03-07T18:01:53.680391Z", "running": 0, "size": 1}}] 2022-03-07T18:01:53.894 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T18:01:53.894 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T18:01:53.895 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-07T18:01:53.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-07T18:01:53.937 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-07T18:01:53.953 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-07T18:01:53.972 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T18:01:53.972 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T18:01:53.991 DEBUG:teuthology.orchestra.remote:smithi057:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-07T18:01:54.003 DEBUG:teuthology.orchestra.run.smithi057:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-07T18:01:54.068 INFO:teuthology.orchestra.run.smithi057.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-07T18:01:54.088 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_27130916b2a523fa33b0cb74b8a8ba8c5a58e8f2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T18:01:54.089 DEBUG:teuthology.orchestra.run.smithi057:> kubectl delete -f cluster.yaml 2022-03-07T18:01:54.246 INFO:teuthology.orchestra.run.smithi057.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-07T18:01:54.254 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:01:54.242170 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-07T18:01:54.524 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:01:54.516582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:01:54.870 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:01:54.859788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:01:54.870 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:01:54.859858 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:05.089 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:05.081975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:05.104 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:05.095844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:15.332 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:15.324575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:15.351 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:15.343784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:25.571 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:25.563376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:25.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:25.579998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:35.806 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:35.798605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:35.823 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:35.812815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:46.045 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:46.037284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:46.059 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:46.052185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:56.282 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:56.274122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:02:56.295 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:02:56.287996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:06.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:06.509864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:06.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:06.525123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:16.757 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:16.749873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:16.771 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:16.764479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:26.998 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:26.990416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:27.012 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:27.005333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:37.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:37.227714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:37.247 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:37.239708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:47.470 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:47.463527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:47.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:47.476614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:57.710 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:57.703771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:03:57.722 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:03:57.716036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:07.946 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:07.939169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:07.969 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:07.962413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:18.203 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:18.196784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:18.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:18.208518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:28.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:28.430405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:28.504 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:28.498254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:38.776 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:38.771038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:38.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:38.783807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:49.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:49.019712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:49.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:49.032782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:59.256 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:59.251024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:04:59.271 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:04:59.266167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:09.492 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:09.487142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:09.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:09.503753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:19.731 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:19.726454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:19.744 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:19.740545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:29.963 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:29.958796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:29.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:29.982954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:40.249 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:40.245255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:40.268 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:40.264908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:50.488 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:50.484040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:05:50.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:05:50.497880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:00.718 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:00.714992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:00.764 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:00.761075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:11.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:11.023794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:11.041 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:11.038521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:21.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:21.322974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:21.342 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:21.338712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:31.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:31.585281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:31.602 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:31.599813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:41.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:41.870263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:41.888 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:41.885247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:52.129 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:52.126866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:06:52.145 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:06:52.142556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:02.403 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:02.400613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:02.418 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:02.415918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:12.645 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:12.643049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:12.659 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:12.657154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:22.878 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:22.876560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:22.896 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:22.892710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:33.115 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:33.113119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:33.128 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:33.126901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:43.349 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:43.347664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:43.363 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:43.362439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:53.607 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:53.605946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:53.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:53.617353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:07:53.678 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:53.677165 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 18:07:53.628566817 +0000 UTC m=+1893.937910913 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T18:07:54.924 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:54.922545 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T18:07:55.566 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:55.564452 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:07:55.554+0000 7efe0164a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:07:55.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:07:55.554+0000 7efe0164a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:07:55.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:07:55.554+0000 7efe0164a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:07:55.567 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:07:55.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:07:55.564481 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:07:55.554+0000 7efe0164a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:07:55.568 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:07:55.554+0000 7efe0164a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:07:55.568 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:07:55.554+0000 7efe0164a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:07:55.568 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:08:03.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:03.837550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:03.853 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:03.852049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:14.079 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:14.078219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:14.125 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:14.124202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:24.354 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:24.353504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:24.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:24.410957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:34.641 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:34.640626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:34.656 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:34.655960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:44.885 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:44.884500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:44.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:44.900212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:55.126 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:55.125478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:08:55.140 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:08:55.140078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:05.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:05.414436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:05.429 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:05.429461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:15.651 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:15.651462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:15.667 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:15.668142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:25.889 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:25.889440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:25.930 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:25.930274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:36.184 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:36.184691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:36.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:36.241039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:46.474 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:46.474894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:46.485 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:46.486188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:56.705 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:56.705751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:09:56.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:09:56.744431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:07.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:07.000678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:07.013 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:07.014476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:17.231 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:17.231919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:17.246 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:17.246827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:27.476 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:27.477072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:27.491 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:27.492469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:37.718 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:37.719592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:37.733 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:37.734550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:47.956 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:47.957541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:47.974 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:47.975036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:58.196 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:58.198203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:10:58.213 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:10:58.214467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:08.515 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:08.517162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:08.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:08.533765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:18.751 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:18.752521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:18.766 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:18.767727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:28.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:28.992429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:29.006 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:29.008269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:39.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:39.237573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:39.250 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:39.252379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:49.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:49.480403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:49.493 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:49.495650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:59.720 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:59.722386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:11:59.737 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:11:59.739856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:09.959 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:09.961168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:09.978 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:09.980758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:20.196 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:20.198177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:20.278 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:20.280195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:30.497 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:30.499779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:30.555 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:30.557593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:40.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:40.779081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:40.796 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:40.799009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:51.017 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:51.020466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:12:51.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:12:51.082594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:01.301 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:01.303907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:01.316 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:01.318113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:11.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:11.585212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:11.600 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:11.603000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:21.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:21.827375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:21.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:21.841934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:32.062 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:32.065067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:32.078 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:32.080847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:42.299 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:42.302299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:42.345 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:42.348150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:52.575 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:52.578955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:13:52.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:13:52.591626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:02.828 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:02.831221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:02.840 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:02.843488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:13.104 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:13.108298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:13.121 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:13.125465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:23.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:23.351019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:23.364 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:23.365544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:33.589 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:33.592772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:33.603 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:33.607443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:43.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:43.830367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:43.840 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:43.844424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:54.064 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:54.068690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:14:54.078 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:14:54.082934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:04.325 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:04.329762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:04.371 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:04.375352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:14.639 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:14.643500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:14.653 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:14.657756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:24.880 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:24.884281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:24.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:24.900052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:35.164 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:35.167852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:35.178 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:35.182959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:45.460 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:45.464405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:45.475 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:45.479538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:55.732 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:55.736604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:15:55.745 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:15:55.749522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:05.973 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:05.978120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:05.984 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:05.989098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:16.206 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:16.210448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:16.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:16.264333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:26.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:26.488452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:26.493 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:26.498592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:36.715 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:36.720739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:36.731 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:36.735676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:46.961 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:46.966636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:46.973 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:46.978362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:57.220 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:57.225460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:16:57.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:16:57.240798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:07.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:07.506770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:07.516 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:07.521977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:17.734 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:17.740037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:17.795 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:17.797632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:28.012 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:28.017870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:28.053 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:28.058435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:38.270 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:38.275242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:38.289 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:38.289847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:48.579 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:48.584102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:48.593 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:48.599185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:58.813 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:58.819120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:17:58.830 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:17:58.835067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:09.051 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:09.056797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:09.066 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:09.071948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:19.285 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:19.291576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:19.299 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:19.305687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:29.520 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:29.525946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:29.533 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:29.538999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:39.754 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:39.759742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:39.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:39.774480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:49.988 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:49.994488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:18:50.006 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:18:50.009171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:00.234 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:00.240083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:00.250 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:00.256170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:10.467 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:10.473220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:10.482 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:10.488081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:20.707 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:20.712949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:20.724 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:20.730422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:30.979 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:30.984787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:30.995 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:31.001954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:41.217 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:41.223553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:41.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:41.239937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:51.454 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:51.460468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:19:51.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:19:51.475698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:01.689 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:01.695854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:01.705 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:01.709910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:11.927 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:11.933117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:11.940 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:11.946433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:22.163 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:22.169395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:22.176 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:22.183247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:32.397 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:32.403805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:32.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:32.418153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:42.630 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:42.636750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:42.644 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:42.650915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:52.867 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:52.874169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:20:52.883 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:20:52.889901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:03.108 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:03.115446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:03.123 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:03.130096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:13.346 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:13.352962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:13.363 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:13.370482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:23.584 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:23.590783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:23.600 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:23.607610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:33.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:33.846018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:33.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:33.862532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:44.074 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:44.080693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:44.087 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:44.094951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:54.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:54.311659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:21:54.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:21:54.327583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:04.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:04.555322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:04.564 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:04.571440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:14.787 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:14.794437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:14.802 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:14.809512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:25.030 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:25.037562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:25.047 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:25.054078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:35.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:35.286463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:35.296 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:35.303918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:45.519 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:45.526679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:45.533 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:45.540920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:55.758 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:55.762979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:22:55.775 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:22:55.773812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:05.988 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:05.995524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:06.002 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:06.010070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:16.227 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:16.234265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:16.245 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:16.251938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:26.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:26.485436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:26.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:26.502680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:36.721 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:36.728848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:36.735 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:36.743341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:46.954 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:46.961763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:46.968 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:46.976048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:57.231 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:57.238880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:23:57.244 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:23:57.251461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:07.470 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:07.477386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:07.484 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:07.491776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:17.710 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:17.717338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:17.764 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:17.771639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:27.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:27.994821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:28.001 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:28.009126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:35.565 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:35.573408 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T18:24:36.229 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:36.236324 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:24:36.223+0000 7f20d9e70340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:24:36.230 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:24:36.223+0000 7f20d9e70340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:24:36.230 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:24:36.223+0000 7f20d9e70340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:24:36.230 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:24:36.230 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:36.236356 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:24:36.223+0000 7f20d9e70340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:24:36.231 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:24:36.223+0000 7f20d9e70340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:24:36.231 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:24:36.223+0000 7f20d9e70340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:24:36.231 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:24:38.229 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:38.237095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:38.246 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:38.253661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:48.471 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:48.478339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:48.519 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:48.523793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:58.740 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:58.748145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:24:58.757 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:24:58.764200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:08.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:08.999226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:09.004 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:09.012794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:09.071 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:09.078906 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 18:25:09.067808126 +0000 UTC m=+2929.377152236 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T18:25:19.226 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:19.233945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:19.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:19.248446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:29.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:29.470956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:29.477 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:29.485463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:39.705 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:39.713276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:39.717 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:39.725914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:49.933 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:49.941222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:25:49.947 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:25:49.955766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:00.201 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:00.208110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:00.253 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:00.260019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:10.474 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:10.481927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:10.487 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:10.495700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:20.714 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:20.721766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:20.727 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:20.735055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:30.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:30.959491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:30.966 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:30.974632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:41.190 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:41.198082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:41.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:41.213129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:51.425 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:51.433103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:26:51.440 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:26:51.448498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:01.661 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:01.669845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:01.674 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:01.682929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:11.894 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:11.902105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:11.910 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:11.918330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:22.136 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:22.144298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:22.150 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:22.158724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:32.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:32.386332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:32.391 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:32.399504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:42.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:42.624261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:42.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:42.639887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:52.866 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:52.874256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:27:52.880 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:27:52.888866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:03.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:03.107931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:03.114 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:03.123163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:13.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:13.423916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:13.430 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:13.439302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:23.655 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:23.664150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:23.667 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:23.676543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:33.909 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:33.917874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:33.921 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:33.929885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:44.143 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:44.151603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:44.162 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:44.170461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:54.405 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:54.413875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:28:54.418 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:28:54.427039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:04.642 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:04.650691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:04.684 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:04.692687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:14.905 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:14.913577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:14.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:14.924549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:25.159 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:25.168344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:25.176 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:25.185303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:35.400 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:35.409440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:35.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:35.424115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:45.636 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:45.645562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:45.647 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:45.656460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:55.871 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:55.879689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:29:55.885 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:29:55.894955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:06.120 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:06.128727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:06.167 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:06.176228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:16.389 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:16.398256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:16.404 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:16.413390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:26.628 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:26.637144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:26.643 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:26.652668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:36.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:36.882401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:36.889 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:36.898923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:47.114 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:47.123688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:47.129 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:47.138814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:57.352 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:57.360968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:30:57.370 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:30:57.376528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:07.589 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:07.598177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:07.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:07.614269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:17.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:17.835891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:17.842 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:17.852359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:28.065 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:28.074527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:28.075 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:28.085392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:38.298 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:38.307437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:38.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:38.321490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:48.535 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:48.544371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:48.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:48.558377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:58.769 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:58.778562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:31:58.786 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:31:58.796361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:09.010 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:09.019993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:09.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:09.035655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:19.254 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:19.263598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:19.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:19.277203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:29.540 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:29.549578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:29.562 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:29.572312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:39.800 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:39.809203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:39.812 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:39.821992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:50.035 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:50.044846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:32:50.064 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:32:50.073212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:00.289 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:00.298961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:00.303 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:00.311934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:10.540 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:10.550122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:10.551 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:10.561203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:20.806 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:20.815958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:20.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:20.831934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:31.039 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:31.048809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:31.055 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:31.064929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:41.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:41.289070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:41.295 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:41.305815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:51.523 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:51.532883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:33:51.534 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:33:51.544782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:01.753 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:01.762541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:01.763 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:01.773502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:11.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:11.997568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:12.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:12.010963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:22.226 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:22.236271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:22.240 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:22.250358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:32.502 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:32.512675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:32.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:32.527605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:42.793 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:42.802978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:42.807 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:42.817890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:53.032 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:53.042393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:34:53.046 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:34:53.057025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:03.271 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:03.280821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:03.286 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:03.296767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:13.509 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:13.519572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:13.526 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:13.536378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:23.759 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:23.769675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:23.772 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:23.782903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:34.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:34.036072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:34.040 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:34.050110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:44.266 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:44.275787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:44.280 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:44.290691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:54.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:54.511476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:35:54.584 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:35:54.590564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:04.805 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:04.815158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:04.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:04.828580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:15.043 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:15.053129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:15.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:15.101419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:25.318 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:25.328626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:25.334 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:25.344260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:35.579 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:35.588953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:35.592 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:35.602808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:45.813 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:45.823702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:45.865 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:45.875912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:56.105 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:56.115218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:36:56.120 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:36:56.131227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:06.368 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:06.378083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:06.381 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:06.391663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:16.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:16.629327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:16.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:16.641590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:26.925 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:26.935565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:26.939 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:26.950002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:37.162 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:37.172921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:37.174 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:37.184670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:47.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:47.410056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:47.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:47.424005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:57.646 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:57.656408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:37:57.662 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:37:57.672405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:07.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:07.902430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:07.907 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:07.917416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:18.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:18.141530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:18.266 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:18.276758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:28.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:28.555193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:28.559 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:28.569256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:38.798 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:38.808002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:38.812 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:38.821663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:49.030 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:49.039636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:49.047 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:49.056484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:59.277 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:59.286201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:38:59.293 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:38:59.302388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:09.519 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:09.528198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:09.536 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:09.545921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:19.759 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:19.768090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:19.772 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:19.781386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:29.990 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:29.998997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:30.003 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:30.012857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:40.240 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:40.248849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:40.254 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:40.263704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:50.535 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:50.544358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:39:50.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:39:50.555043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:00.779 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:00.788143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:00.795 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:00.803947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:11.113 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:11.121742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:11.123 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:11.132134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:21.398 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:21.406246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:21.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:21.422411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:31.636 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:31.643714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:31.650 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:31.658694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:41.872 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:41.880147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:41.927 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:41.935786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:52.156 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:52.163861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:40:52.171 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:40:52.179499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:02.394 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:02.402247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:02.452 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:02.460521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:12.675 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:12.683212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:12.694 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:12.702105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:16.239 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:16.247318 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T18:41:16.955 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:16.961934 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:41:16.947+0000 7f9a23e7e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:41:16.955 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:41:16.947+0000 7f9a23e7e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:41:16.955 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:41:16.947+0000 7f9a23e7e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:41:16.956 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:41:16.956 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:16.961966 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:41:16.947+0000 7f9a23e7e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:41:16.956 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:41:16.947+0000 7f9a23e7e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:41:16.956 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:41:16.947+0000 7f9a23e7e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:41:16.957 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:41:22.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:22.922772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:22.931 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:22.938769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:33.190 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:33.197783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:33.211 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:33.215490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:43.430 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:43.437422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:43.446 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:43.453601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:53.673 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:53.680453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:41:53.703 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:41:53.693914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:03.907 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:03.914555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:03.950 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:03.954718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:14.202 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:14.209140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:14.217 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:14.224638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:24.438 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:24.445609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:24.448 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:24.455840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:24.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:24.564670 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 18:42:24.541898554 +0000 UTC m=+3964.851242650 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T18:42:34.670 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:34.677046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:34.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:34.693220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:44.908 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:44.914816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:44.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:44.948223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:55.157 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:55.163358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:42:55.171 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:42:55.178287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:05.441 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:05.447507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:05.456 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:05.462594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:15.684 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:15.690371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:15.699 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:15.706287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:25.922 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:25.928844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:25.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:25.947880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:36.192 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:36.198729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:36.207 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:36.213674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:46.429 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:46.435992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:46.440 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:46.446188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:56.663 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:56.668966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:43:56.677 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:43:56.684155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:06.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:06.905997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:06.921 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:06.923540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:17.191 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:17.197395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:17.209 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:17.215369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:27.462 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:27.467817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:27.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:27.523165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:37.738 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:37.744174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:37.754 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:37.759921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:47.997 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:48.003425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:48.014 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:48.020789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:58.235 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:58.240832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:44:58.250 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:44:58.255964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:08.477 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:08.482629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:08.503 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:08.509292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:18.725 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:18.730666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:18.741 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:18.746594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:28.960 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:28.966404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:28.972 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:28.978474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:39.191 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:39.196589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:39.207 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:39.212885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:49.432 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:49.437300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:49.445 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:49.451487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:59.669 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:59.674881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:45:59.680 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:45:59.685806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:09.897 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:09.902971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:09.948 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:09.953892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:20.173 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:20.178869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:20.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:20.192520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:30.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:30.416786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:30.431 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:30.435885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:40.649 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:40.654493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:40.663 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:40.669149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:50.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:50.896646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:46:50.908 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:46:50.912859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:01.137 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:01.142358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:01.154 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:01.159064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:11.404 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:11.408933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:11.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:11.418961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:21.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:21.659083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:21.673 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:21.678290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:31.896 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:31.900856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:31.910 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:31.915172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:42.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:42.137086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:42.150 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:42.154921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:52.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:52.383434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:47:52.396 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:47:52.400773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:02.617 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:02.622568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:02.632 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:02.636912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:12.864 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:12.869101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:12.916 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:12.920503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:23.136 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:23.141032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:23.152 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:23.156765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:33.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:33.382663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:33.392 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:33.396711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:43.614 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:43.618764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:43.629 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:43.633905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:53.882 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:53.887043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:48:53.902 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:48:53.906729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:04.158 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:04.162544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:04.173 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:04.177690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:14.470 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:14.474673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:14.533 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:14.534580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:24.770 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:24.774651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:24.784 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:24.788516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:35.009 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:35.014074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:35.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:35.030233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:45.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:45.271645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:45.281 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:45.285530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:55.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:55.509362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:49:55.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:49:55.521360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:05.737 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:05.741113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:05.750 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:05.754820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:15.972 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:15.976387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:15.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:15.991075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:26.209 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:26.213629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:26.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:26.227422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:36.446 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:36.450510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:36.460 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:36.464746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:46.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:46.689758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:46.703 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:46.706054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:56.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:56.932070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:50:56.942 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:50:56.946435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:07.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:07.173529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:07.183 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:07.187640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:17.423 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:17.427148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:17.437 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:17.441966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:27.662 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:27.666236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:27.677 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:27.681175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:37.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:37.899239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:37.909 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:37.913358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:48.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:48.138349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:48.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:48.152793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:58.369 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:58.373673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:51:58.385 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:51:58.389136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:08.646 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:08.650381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:08.659 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:08.663652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:18.886 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:18.890453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:18.902 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:18.906342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:29.119 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:29.123038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:29.135 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:29.138765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:39.361 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:39.364593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:39.376 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:39.379946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:49.597 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:49.600572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:49.616 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:49.620051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:59.832 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:59.835824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:52:59.848 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:52:59.852115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:10.071 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:10.074444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:10.090 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:10.093681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:20.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:20.322818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:20.332 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:20.336124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:30.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:30.561604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:30.570 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:30.574038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:40.791 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:40.793651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:40.805 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:40.808585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:51.035 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:51.039168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:53:51.052 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:53:51.055943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:01.304 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:01.307423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:01.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:01.322746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:11.540 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:11.543565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:11.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:11.560532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:21.780 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:21.783797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:21.794 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:21.798232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:32.018 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:32.021885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:32.033 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:32.036303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:42.257 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:42.260728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:42.272 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:42.276140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:52.500 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:52.504125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:54:52.522 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:54:52.522503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:02.741 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:02.744594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:02.756 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:02.760310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:12.974 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:12.977632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:12.993 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:12.996193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:23.213 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:23.216970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:23.232 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:23.235545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:33.459 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:33.462629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:33.475 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:33.478246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:43.730 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:43.733059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:43.744 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:43.748133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:53.966 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:53.969369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:55:53.978 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:55:53.982168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:04.209 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:04.212678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:04.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:04.226787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:14.453 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:14.455732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:14.465 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:14.468993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:24.699 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:24.701341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:24.714 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:24.717939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:34.935 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:34.938733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:34.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:34.954696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:45.172 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:45.175152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:45.184 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:45.188135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:55.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:55.472427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:56:55.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:56:55.486846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:05.708 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:05.711365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:05.721 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:05.723854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:15.942 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:15.944330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:15.956 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:15.959846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:26.181 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:26.184631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:26.197 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:26.199959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:36.419 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:36.422640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:36.436 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:36.439860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:46.657 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:46.659862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:46.673 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:46.675915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:56.899 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:56.901746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:56.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:56.917963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:57:57.096 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:57.099093 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T18:57:57.820 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:57.822299 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:57:57.811+0000 7f8eaa74c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:57:57.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:57:57.811+0000 7f8eaa74c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:57:57.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:57:57.811+0000 7f8eaa74c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:57:57.821 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:57:57.821 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:57:57.822339 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T18:57:57.811+0000 7f8eaa74c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T18:57:57.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:57:57.811+0000 7f8eaa74c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T18:57:57.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T18:57:57.811+0000 7f8eaa74c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T18:57:57.822 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T18:58:07.145 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:07.148206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:07.161 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:07.164197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:17.404 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:17.407202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:17.417 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:17.420322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:27.640 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:27.643475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:27.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:27.841777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:38.066 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:38.069131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:38.082 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:38.084506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:48.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:48.307753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:48.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:48.322129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:58.595 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:58.598696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:58:58.609 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:58:58.612056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:08.848 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:08.850873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:08.862 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:08.865700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:19.119 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:19.122042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:19.208 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:19.210859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:29.486 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:29.488857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:29.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:29.547637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:39.782 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:39.785188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:39.797 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:39.800119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:40.087 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:40.090760 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 18:59:40.052165285 +0000 UTC m=+5000.361509390 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T18:59:50.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:50.019525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T18:59:50.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 18:59:50.034434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:00.251 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:00.254050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:00.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:00.270057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:10.486 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:10.489096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:10.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:10.504324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:20.745 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:20.748208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:20.755 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:20.758832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:31.002 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:31.004822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:31.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:31.018980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:41.242 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:41.245481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:41.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:41.261806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:51.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:51.511188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:00:51.522 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:00:51.525708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:01.746 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:01.748945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:01.762 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:01.765142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:11.984 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:11.987000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:12.034 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:12.037132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:22.252 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:22.254919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:22.271 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:22.272057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:32.492 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:32.495242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:32.511 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:32.510665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:42.732 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:42.734948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:42.746 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:42.749215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:52.970 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:52.972646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:01:52.992 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:01:52.994228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:03.222 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:03.225243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:03.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:03.243891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:03.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:03.243926 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:13.475 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:13.477542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:13.489 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:13.492028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:23.755 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:23.757635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:23.770 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:23.772568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:34.010 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:34.012919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:34.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:34.027845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:44.249 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:44.251587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:44.264 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:44.267169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:54.486 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:54.488786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:02:54.502 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:02:54.504621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:04.729 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:04.732235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:04.746 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:04.749512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:15.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:15.471607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:15.487 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:15.489974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:25.707 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:25.709957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:25.719 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:25.722716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:36.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:36.040550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:36.052 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:36.054772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:46.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:46.269859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:46.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:46.282562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:56.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:56.498548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:03:56.510 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:03:56.511205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:06.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:06.822041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:06.834 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:06.837638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:17.061 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:17.064515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:17.074 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:17.077192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:27.292 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:27.294496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:27.308 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:27.310892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:37.556 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:37.558936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:37.572 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:37.574839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:47.791 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:47.794081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:47.806 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:47.809437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:58.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:58.034101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:04:58.047 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:04:58.050401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:08.299 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:08.301747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:08.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:08.316025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:18.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:18.532621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:18.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:18.548146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:28.774 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:28.776720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:28.831 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:28.833415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:39.054 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:39.056471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:39.068 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:39.071304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:49.290 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:49.292582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:49.304 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:49.306637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:59.524 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:59.526856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:05:59.543 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:05:59.544422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:09.762 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:09.765336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:09.779 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:09.781758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:20.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:20.002609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:20.012 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:20.014817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:30.231 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:30.233614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:30.245 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:30.248015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:40.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:40.471430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:40.486 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:40.488773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:50.708 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:50.710699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:06:50.723 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:06:50.726092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:00.946 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:00.948516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:00.961 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:00.963667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:11.266 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:11.268492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:11.277 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:11.279789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:21.498 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:21.500711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:21.514 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:21.516604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:31.742 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:31.744907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:31.757 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:31.757964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:41.989 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:41.991780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:42.006 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:42.009114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:52.268 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:52.270846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:07:52.284 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:07:52.285184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:02.505 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:02.507902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:02.521 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:02.524074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:12.738 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:12.741076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:12.755 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:12.758101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:22.978 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:22.980436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:22.993 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:22.996052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:33.227 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:33.230198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:33.242 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:33.245266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:43.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:43.471901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:43.482 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:43.485122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:53.702 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:53.704554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:08:53.712 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:08:53.714917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:03.935 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:03.937666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:03.949 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:03.951602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:14.176 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:14.178826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:14.192 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:14.194728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:24.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:24.417530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:24.429 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:24.431541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:34.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:34.657101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:34.670 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:34.672281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:44.902 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:44.904227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:44.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:44.918302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:55.181 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:55.184057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:09:55.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:09:55.201551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:05.420 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:05.423217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:05.434 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:05.436799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:15.657 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:15.660004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:15.672 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:15.674843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:25.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:25.894491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:25.904 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:25.906713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:36.124 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:36.125933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:36.137 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:36.139558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:46.357 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:46.359443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:46.374 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:46.374697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:56.594 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:56.596293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:10:56.608 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:10:56.610397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:06.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:06.828350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:06.843 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:06.844675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:17.057 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:17.059759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:17.076 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:17.079344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:27.299 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:27.301381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:27.314 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:27.316883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:37.549 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:37.552001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:37.561 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:37.563752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:47.783 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:47.785686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:47.797 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:47.799622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:58.037 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:58.039903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:11:58.049 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:11:58.052032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:08.282 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:08.284037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:08.297 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:08.299487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:18.543 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:18.545359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:18.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:18.559919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:28.781 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:28.782884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:28.794 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:28.794686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:39.013 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:39.014546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:39.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:39.030437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:49.255 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:49.256980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:49.273 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:49.274711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:59.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:59.496421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:12:59.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:12:59.509590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:09.735 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:09.737162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:09.745 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:09.746823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:19.979 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:19.977082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:19.990 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:19.991836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:30.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:30.215368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:30.227 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:30.229165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:40.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:40.484799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:40.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:40.502494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:50.725 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:50.725738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:13:50.740 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:13:50.741802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:00.988 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:00.988074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:01.002 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:01.003570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:11.226 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:11.226848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:11.256 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:11.256078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:21.492 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:21.493072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:21.505 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:21.506010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:31.729 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:31.730101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:31.745 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:31.745482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:37.843 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:37.843379 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T19:14:38.550 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:38.550111 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T19:14:38.539+0000 7fbe65ce7340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T19:14:38.551 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:14:38.539+0000 7fbe65ce7340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T19:14:38.551 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:14:38.539+0000 7fbe65ce7340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T19:14:38.551 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T19:14:38.551 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:38.550152 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T19:14:38.539+0000 7fbe65ce7340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T19:14:38.552 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:14:38.539+0000 7fbe65ce7340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T19:14:38.552 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:14:38.539+0000 7fbe65ce7340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T19:14:38.552 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T19:14:41.965 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:41.966003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:41.975 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:41.976718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:52.212 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:52.212497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:14:52.228 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:14:52.229464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:02.447 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:02.447887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:02.460 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:02.461109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:12.679 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:12.680192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:12.693 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:12.694177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:22.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:22.910931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:22.923 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:22.923658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:33.139 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:33.139446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:33.154 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:33.154584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:43.407 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:43.407501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:43.419 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:43.419525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:53.639 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:53.639049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:15:53.655 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:15:53.654452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:03.878 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:03.878773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:03.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:03.892957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:14.116 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:14.116313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:14.171 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:14.168859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:24.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:24.483057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:24.511 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:24.511502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:34.736 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:34.735911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:34.753 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:34.752865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:44.975 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:44.974246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:44.988 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:44.988356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:55.209 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:55.208913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:55.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:55.223201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:16:55.253 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:16:55.253634 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 19:16:55.239576188 +0000 UTC m=+6035.548920258 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T19:17:05.449 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:05.448719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:05.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:05.463320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:15.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:15.685368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:15.701 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:15.700384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:25.925 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:25.924018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:25.940 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:25.939164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:36.164 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:36.162863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:36.177 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:36.177179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:46.403 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:46.402613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:46.422 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:46.418815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:56.639 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:56.638414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:17:56.653 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:17:56.652930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:06.877 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:06.876023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:06.888 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:06.887357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:17.116 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:17.115069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:17.130 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:17.129585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:27.357 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:27.356028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:27.372 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:27.371463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:37.605 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:37.603564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:37.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:37.616955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:47.837 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:47.836011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:47.852 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:47.850493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:58.110 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:58.108385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:18:58.127 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:18:58.126040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:08.348 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:08.347234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:08.371 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:08.369968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:18.632 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:18.630679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:18.647 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:18.646715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:28.904 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:28.902718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:28.919 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:28.917998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:39.140 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:39.139228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:39.154 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:39.153203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:49.377 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:49.375736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:49.391 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:49.390073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:59.613 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:59.610955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:19:59.629 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:19:59.627810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:09.854 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:09.852421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:09.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:09.867482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:20.094 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:20.092715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:20.114 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:20.112488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:30.341 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:30.338986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:30.354 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:30.353095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:40.572 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:40.570258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:40.585 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:40.584126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:50.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:50.814875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:20:50.828 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:20:50.826000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:01.050 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:01.047585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:01.063 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:01.061752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:11.288 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:11.285666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:11.302 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:11.300845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:21.531 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:21.529113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:21.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:21.544120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:31.774 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:31.771906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:31.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:31.786876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:42.017 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:42.015012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:42.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:42.029238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:52.256 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:52.253937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:21:52.275 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:21:52.273835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:02.505 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:02.497421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:02.511 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:02.509673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:12.736 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:12.734035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:12.751 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:12.748749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:22.977 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:22.974920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:22.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:22.989385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:33.220 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:33.217416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:33.233 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:33.231195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:43.492 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:43.490201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:43.508 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:43.506679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:53.750 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:53.748116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:22:53.767 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:22:53.763606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:04.005 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:03.995748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:04.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:04.007938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:14.239 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:14.236802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:14.255 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:14.252065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:24.482 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:24.479078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:24.496 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:24.493235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:34.721 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:34.716968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:34.740 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:34.737776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:44.964 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:44.961787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:44.979 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:44.976968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:55.206 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:55.202604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:23:55.220 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:23:55.217544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:05.441 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:05.438951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:05.459 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:05.456242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:15.682 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:15.679209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:15.698 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:15.695847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:25.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:25.925488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:25.944 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:25.941772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:36.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:36.166291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:36.185 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:36.182835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:46.412 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:46.409017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:46.429 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:46.426979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:56.655 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:56.652528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:24:56.672 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:24:56.669254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:06.922 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:06.919156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:06.934 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:06.931177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:17.154 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:17.150277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:17.167 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:17.164185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:27.393 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:27.389956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:27.408 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:27.405376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:37.634 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:37.630285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:37.658 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:37.655301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:47.881 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:47.877997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:47.902 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:47.899215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:58.129 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:58.126432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:25:58.142 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:25:58.138996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:08.362 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:08.358675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:08.380 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:08.377169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:18.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:18.599339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:18.617 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:18.614077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:28.850 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:28.846840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:28.860 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:28.857538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:39.088 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:39.084212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:39.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:39.097715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:49.325 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:49.320854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:49.338 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:49.334881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:59.581 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:59.578349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:26:59.635 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:26:59.631519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:09.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:09.865971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:09.890 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:09.884895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:20.128 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:20.124897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:20.144 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:20.140845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:30.382 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:30.378004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:30.396 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:30.393017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:40.648 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:40.644441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:40.661 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:40.657873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:50.883 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:50.880115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:27:50.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:27:50.896672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:01.122 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:01.118826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:01.135 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:01.132359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:11.358 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:11.354549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:11.372 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:11.368697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:21.599 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:21.595067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:21.613 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:21.610022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:31.842 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:31.838713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:31.858 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:31.854817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:42.119 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:42.115314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:42.133 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:42.129813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:52.359 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:52.354764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:28:52.373 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:28:52.370074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:02.602 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:02.598198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:02.622 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:02.615830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:12.843 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:12.839516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:12.872 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:12.869006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:23.093 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:23.089676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:23.108 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:23.104430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:33.331 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:33.327078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:33.348 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:33.344893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:43.569 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:43.565163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:43.580 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:43.577456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:53.806 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:53.802350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:29:53.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:29:53.817306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:04.044 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:04.040405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:04.058 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:04.055042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:14.282 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:14.278456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:14.299 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:14.295818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:24.550 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:24.546572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:24.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:24.562394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:34.788 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:34.784739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:34.805 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:34.801804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:45.032 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:45.028743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:45.046 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:45.042917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:55.268 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:55.264241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:30:55.282 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:30:55.278258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:05.539 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:05.534889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:05.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:05.550186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:15.810 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:15.805740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:15.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:15.821222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:18.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:18.555906 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T19:31:19.204 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:19.198952 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T19:31:19.187+0000 7f1d61872340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T19:31:19.204 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:31:19.187+0000 7f1d61872340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T19:31:19.204 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:31:19.187+0000 7f1d61872340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T19:31:19.205 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T19:31:19.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:19.198993 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T19:31:19.187+0000 7f1d61872340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T19:31:19.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:31:19.187+0000 7f1d61872340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T19:31:19.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:31:19.187+0000 7f1d61872340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T19:31:19.206 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T19:31:26.055 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:26.051362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:26.072 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:26.068075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:36.355 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:36.351657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:36.374 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:36.368516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:46.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:46.614241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:46.635 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:46.630881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:56.861 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:56.856917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:31:56.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:31:56.872100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:07.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:07.096831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:07.112 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:07.108566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:17.335 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:17.330501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:17.349 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:17.345495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:27.639 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:27.634741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:27.651 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:27.647643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:37.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:37.872043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:37.891 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:37.886685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:48.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:48.113365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:48.130 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:48.126472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:58.352 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:58.348038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:32:58.367 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:32:58.363480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:08.589 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:08.585476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:08.605 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:08.601240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:18.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:18.818534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:18.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:18.869604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:29.095 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:29.091521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:29.111 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:29.105689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:39.337 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:39.332709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:39.355 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:39.350875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:49.576 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:49.572449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:49.592 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:49.587971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:59.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:59.814375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:33:59.833 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:33:59.828849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:10.056 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:10.051929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:10.070 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:10.065329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:20.291 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:20.287150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:20.308 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:20.303553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:30.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:30.527182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:30.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:30.541051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:40.767 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:40.763122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:40.783 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:40.778738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:51.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:51.011222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:34:51.032 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:34:51.026287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:01.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:01.254417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:01.272 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:01.267698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:11.527 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:11.522444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:11.540 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:11.536207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:11.740 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:11.735653 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 19:35:11.724699183 +0000 UTC m=+7132.034043285 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T19:35:21.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:21.782424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:21.802 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:21.798182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:32.055 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:32.050333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:32.069 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:32.065193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:42.293 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:42.288356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:42.358 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:42.354316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:52.605 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:52.601012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:35:52.621 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:35:52.617474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:02.903 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:02.898581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:02.916 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:02.912434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:13.137 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:13.133056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:13.154 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:13.149943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:23.372 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:23.368233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:23.431 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:23.424452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:33.652 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:33.648048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:33.668 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:33.663900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:43.882 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:43.878072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:43.896 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:43.892315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:54.114 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:54.109748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:36:54.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:36:54.127986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:04.357 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:04.352177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:04.372 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:04.368536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:14.597 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:14.592702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:14.614 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:14.610162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:24.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:24.833656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:24.850 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:24.845869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:35.069 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:35.064098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:35.084 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:35.079686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:45.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:45.300906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:45.321 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:45.316880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:55.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:55.541264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:37:55.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:37:55.556157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:05.788 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:05.783487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:05.804 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:05.800118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:16.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:16.021509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:16.043 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:16.037625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:26.299 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:26.294683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:26.314 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:26.309206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:36.538 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:36.531942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:36.593 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:36.588546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:46.818 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:46.813468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:46.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:46.869146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:57.122 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:57.116750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:38:57.140 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:38:57.131880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:07.376 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:07.371870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:07.391 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:07.387399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:17.622 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:17.617320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:17.634 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:17.628899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:27.861 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:27.856459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:27.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:27.871925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:38.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:38.096387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:38.120 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:38.116153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:48.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:48.342457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:48.393 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:48.388519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:58.627 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:58.622635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:39:58.640 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:39:58.635393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:08.863 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:08.858639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:08.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:08.871655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:19.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:19.093428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:19.112 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:19.108148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:29.338 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:29.332858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:29.354 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:29.349110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:39.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:39.577492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:39.598 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:39.593083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:49.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:49.834832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:40:49.854 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:40:49.849587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:00.081 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:00.076695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:00.096 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:00.091821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:10.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:10.314707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:10.333 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:10.328385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:20.556 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:20.550984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:20.568 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:20.563248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:30.786 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:30.781185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:30.801 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:30.796245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:41.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:41.021346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:41.041 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:41.036794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:51.271 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:51.265768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:41:51.287 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:41:51.282539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:01.510 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:01.505418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:01.561 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:01.556409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:11.784 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:11.778712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:11.798 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:11.793479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:22.067 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:22.061004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:22.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:22.075903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:32.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:32.306824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:32.329 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:32.324262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:42.549 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:42.544269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:42.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:42.556098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:52.788 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:52.783227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:42:52.799 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:42:52.794636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:03.020 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:03.015354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:03.039 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:03.034428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:13.259 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:13.254621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:13.275 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:13.270513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:23.492 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:23.487867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:23.511 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:23.507010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:33.737 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:33.732567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:33.752 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:33.747175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:44.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:44.009995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:44.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:44.024296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:54.254 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:54.249435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:43:54.270 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:43:54.265807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:04.497 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:04.492628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:04.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:04.512277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:14.744 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:14.739747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:14.807 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:14.802052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:25.044 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:25.039598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:25.059 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:25.054235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:35.284 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:35.279706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:35.298 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:35.293645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:45.533 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:45.528063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:45.547 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:45.542225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:55.798 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:55.793005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:44:55.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:44:55.810174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:06.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:06.032960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:06.052 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:06.047723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:16.270 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:16.265429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:16.281 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:16.276500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:26.504 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:26.499223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:26.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:26.512829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:36.741 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:36.735595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:36.758 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:36.753693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:46.983 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:46.978159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:47.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:46.995315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:57.270 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:57.265579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:45:57.285 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:45:57.278296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:07.514 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:07.509524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:07.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:07.523297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:17.785 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:17.780558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:17.844 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:17.837081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:28.066 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:28.061239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:28.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:28.075436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:38.303 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:38.298293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:38.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:38.314278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:48.541 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:48.535935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:48.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:48.551031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:58.780 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:58.774851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:46:58.794 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:46:58.789831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:09.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:09.011077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:09.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:09.026114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:19.257 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:19.252599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:19.271 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:19.266535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:29.494 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:29.488943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:29.510 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:29.505205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:39.736 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:39.731749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:39.750 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:39.745772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:49.977 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:49.971290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:49.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:49.986362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:47:59.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:59.208929 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T19:47:59.871 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:59.864544 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T19:47:59.852+0000 7f15e1f4b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T19:47:59.871 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:47:59.852+0000 7f15e1f4b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T19:47:59.872 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:47:59.852+0000 7f15e1f4b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T19:47:59.872 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T19:47:59.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:47:59.864575 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T19:47:59.852+0000 7f15e1f4b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T19:47:59.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:47:59.852+0000 7f15e1f4b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T19:47:59.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T19:47:59.852+0000 7f15e1f4b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T19:47:59.874 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T19:48:00.210 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:00.205891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:00.224 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:00.219753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:10.450 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:10.445658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:10.464 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:10.459639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:20.700 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:20.694968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:20.714 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:20.709275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:30.936 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:30.931600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:30.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:30.947074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:41.172 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:41.167461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:41.186 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:41.181681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:51.409 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:51.403956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:48:51.425 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:48:51.420183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:01.648 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:01.643579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:01.666 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:01.662075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:11.888 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:11.883064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:11.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:11.896028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:22.129 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:22.123561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:22.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:22.142227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:32.366 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:32.361185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:32.376 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:32.371769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:42.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:42.594506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:42.611 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:42.606791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:52.832 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:52.827206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:49:52.898 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:49:52.893900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:03.125 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:03.120619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:03.140 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:03.135195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:13.362 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:13.357299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:13.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:13.372934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:23.601 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:23.595855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:23.677 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:23.672966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:33.898 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:33.893707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:33.975 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:33.969566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:44.219 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:44.214274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:44.232 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:44.228053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:54.477 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:54.472468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:50:54.491 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:50:54.486642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:04.713 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:04.708828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:04.731 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:04.725045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:14.956 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:14.951153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:14.972 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:14.967347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:25.195 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:25.190428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:25.208 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:25.203848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:35.433 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:35.429013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:35.448 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:35.443785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:45.671 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:45.666478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:45.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:45.681805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:55.914 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:55.908294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:51:55.931 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:51:55.923468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:06.158 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:06.153067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:06.173 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:06.168845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:16.395 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:16.390677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:16.410 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:16.405482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:26.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:26.626595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:26.645 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:26.640768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:36.883 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:36.877486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:36.909 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:36.904973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:47.179 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:47.174330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:47.194 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:47.189788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:57.417 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:57.412095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:52:57.430 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:52:57.425911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:07.663 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:07.658890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:07.679 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:07.674542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:17.906 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:17.901894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:17.922 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:17.918290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:28.151 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:28.146866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:28.162 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:28.158436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:28.181 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:28.176816 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 19:53:28.167559019 +0000 UTC m=+8228.476903075 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T19:53:38.390 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:38.385709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:38.409 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:38.405188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:48.649 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:48.644421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:48.663 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:48.658922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:58.923 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:58.918940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:53:58.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:53:58.933386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:09.161 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:09.156931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:09.178 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:09.173582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:19.428 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:19.423530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:19.443 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:19.438940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:29.674 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:29.669693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:29.728 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:29.723231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:39.957 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:39.952447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:39.971 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:39.966638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:50.193 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:50.188721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:54:50.209 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:54:50.204959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:00.438 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:00.433677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:00.454 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:00.449774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:10.676 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:10.671949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:10.693 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:10.688449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:20.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:20.911016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:20.929 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:20.924749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:31.157 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:31.152693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:31.172 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:31.168257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:41.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:41.394508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:41.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:41.409726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:51.635 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:51.631102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:55:51.646 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:55:51.642607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:01.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:01.868890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:01.886 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:01.881955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:12.108 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:12.103799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:12.125 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:12.120793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:22.350 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:22.345221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:22.369 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:22.361434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:32.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:32.583606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:32.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:32.598866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:42.829 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:42.824432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:42.843 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:42.839461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:53.066 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:53.061550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:56:53.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:56:53.075872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:03.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:03.300244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:03.320 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:03.315326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:13.569 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:13.564662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:13.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:13.581811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:23.809 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:23.804726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:23.827 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:23.821115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:34.044 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:34.040391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:34.055 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:34.050950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:44.329 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:44.325113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:44.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:44.340987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:54.577 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:54.572753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:57:54.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:57:54.583872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:04.826 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:04.821645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:04.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:04.833763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:15.083 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:15.078210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:15.097 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:15.093394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:25.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:25.314657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:25.332 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:25.327853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:35.550 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:35.546507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:35.565 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:35.560817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:45.824 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:45.819371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:45.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:45.834456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:56.065 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:56.060896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:58:56.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:58:56.076536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:06.309 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:06.304693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:06.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:06.321706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:16.549 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:16.545224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:16.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:16.562705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:26.793 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:26.788750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:26.807 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:26.803444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:37.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:37.026726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:37.043 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:37.038189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:47.309 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:47.305520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:47.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:47.321829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:57.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:57.544329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T19:59:57.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 19:59:57.556534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:07.785 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:07.781012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:07.801 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:07.797429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:18.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:18.020748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:18.038 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:18.034592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:28.262 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:28.257982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:28.277 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:28.273182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:38.507 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:38.502830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:38.522 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:38.518139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:48.750 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:48.745683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:48.765 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:48.761378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:58.988 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:58.982912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:00:58.998 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:00:58.994163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:09.224 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:09.219524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:09.233 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:09.229503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:19.457 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:19.452927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:19.470 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:19.465954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:29.700 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:29.695845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:29.715 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:29.710521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:39.937 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:39.931773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:39.949 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:39.944759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:50.174 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:50.170056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:01:50.188 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:01:50.184579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:00.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:00.411110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:00.435 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:00.427142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:10.705 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:10.701848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:10.739 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:10.732391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:10.739 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:10.732446 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:20.970 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:20.965933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:20.987 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:20.983405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:31.208 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:31.203877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:31.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:31.218419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:41.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:41.479369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:41.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:41.497309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:51.718 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:51.714223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:02:51.731 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:02:51.727722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:01.954 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:01.949586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:01.971 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:01.966857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:12.204 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:12.200476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:12.217 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:12.213473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:22.458 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:22.454013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:22.476 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:22.472237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:32.698 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:32.694206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:32.708 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:32.704209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:42.929 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:42.925110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:42.940 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:42.936646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:53.165 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:53.160578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:03:53.185 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:03:53.175744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:03.446 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:03.441987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:03.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:03.458368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:13.685 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:13.680290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:13.697 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:13.693433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:23.918 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:23.914355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:23.946 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:23.941695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:34.168 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:34.164634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:34.184 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:34.180335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:39.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:39.871908 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T20:04:40.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:40.552252 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:04:40.541+0000 7f34c2acb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:04:40.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:04:40.541+0000 7f34c2acb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:04:40.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:04:40.541+0000 7f34c2acb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:04:40.559 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:04:40.559 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:40.552280 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:04:40.541+0000 7f34c2acb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:04:40.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:04:40.541+0000 7f34c2acb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:04:40.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:04:40.541+0000 7f34c2acb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:04:40.560 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:04:44.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:44.408890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:44.432 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:44.427680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:54.653 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:54.648628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:04:54.669 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:04:54.664205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:04.890 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:04.886174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:04.903 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:04.900099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:15.127 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:15.123176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:15.140 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:15.136562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:25.365 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:25.361400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:25.390 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:25.386174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:35.665 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:35.660821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:36.166 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:35.678274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:45.913 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:45.909328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:45.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:45.924557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:56.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:56.144833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:05:56.167 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:05:56.163466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:06.388 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:06.384498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:06.406 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:06.401771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:16.633 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:16.628680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:16.650 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:16.645415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:26.873 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:26.869203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:26.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:26.924196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:37.152 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:37.148630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:37.167 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:37.163078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:47.392 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:47.388163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:47.408 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:47.404441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:57.632 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:57.628202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:06:57.646 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:06:57.642399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:07.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:07.865678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:07.883 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:07.880005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:18.104 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:18.100539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:18.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:18.113622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:28.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:28.335497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:28.355 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:28.351028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:38.581 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:38.577473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:38.595 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:38.592224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:48.820 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:48.815905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:48.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:48.831437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:59.070 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:59.065665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:07:59.083 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:07:59.079642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:09.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:09.306933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:09.323 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:09.320248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:19.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:19.563580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:19.625 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:19.621755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:29.850 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:29.846721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:29.863 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:29.859470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:40.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:40.087089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:40.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:40.097935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:50.331 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:50.326936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:08:50.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:08:50.343222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:00.564 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:00.560826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:00.578 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:00.574383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:10.804 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:10.800281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:10.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:10.813711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:21.041 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:21.035831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:21.056 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:21.052070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:31.280 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:31.273801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:31.291 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:31.287718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:41.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:41.525949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:41.541 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:41.537216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:51.764 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:51.759969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:09:51.778 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:09:51.774417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:02.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:01.991803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:02.061 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:02.057467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:12.308 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:12.304652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:12.321 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:12.317288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:22.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:22.541605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:22.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:22.554980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:32.791 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:32.787422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:32.806 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:32.802794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:43.099 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:43.094678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:43.115 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:43.112037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:53.341 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:53.337129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:10:53.359 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:10:53.355099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:03.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:03.578655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:03.597 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:03.593610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:13.819 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:13.815779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:13.838 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:13.834725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:24.057 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:24.053124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:24.072 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:24.068447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:34.292 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:34.287993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:34.306 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:34.302205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:44.527 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:44.523371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:44.541 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:44.537709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:44.642 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:44.638316 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 20:11:44.627049944 +0000 UTC m=+9324.936394100 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T20:11:54.764 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:54.760755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:11:54.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:11:54.773855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:05.001 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:04.997088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:05.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:05.012405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:15.241 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:15.236730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:15.255 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:15.251862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:25.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:25.478976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:25.499 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:25.495693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:35.723 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:35.719092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:35.737 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:35.733110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:45.958 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:45.953678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:46.031 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:46.027078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:56.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:56.260012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:12:56.277 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:12:56.274230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:06.526 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:06.521673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:06.541 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:06.537262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:16.763 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:16.759662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:16.779 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:16.775467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:27.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:26.996347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:27.019 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:27.015206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:37.243 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:37.239350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:37.258 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:37.254631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:47.480 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:47.476852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:47.500 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:47.496458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:57.720 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:57.716179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:13:57.735 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:13:57.731123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:07.960 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:07.956233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:07.981 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:07.977183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:18.202 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:18.197888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:18.217 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:18.213332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:28.438 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:28.434926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:28.454 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:28.449765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:38.675 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:38.671224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:38.689 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:38.685289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:48.914 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:48.910010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:48.931 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:48.927375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:59.159 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:59.154966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:14:59.176 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:14:59.169089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:09.402 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:09.398383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:09.416 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:09.412411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:19.687 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:19.681656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:19.702 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:19.695827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:29.921 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:29.917341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:29.935 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:29.932066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:40.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:40.176584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:40.191 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:40.187916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:50.437 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:50.433177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:15:50.447 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:15:50.443854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:00.677 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:00.673832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:00.695 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:00.688269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:10.918 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:10.913925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:10.929 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:10.925892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:21.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:21.164700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:21.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:21.182095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:31.410 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:31.405719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:31.425 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:31.421344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:41.655 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:41.650481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:41.669 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:41.665454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:51.889 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:51.885700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:16:51.903 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:16:51.899364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:02.128 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:02.124509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:02.148 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:02.144493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:12.378 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:12.370742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:12.391 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:12.387087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:22.672 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:22.669114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:22.698 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:22.694587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:32.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:32.924605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:32.947 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:32.943163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:43.178 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:43.174070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:43.191 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:43.188019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:53.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:53.409626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:17:53.430 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:17:53.426868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:03.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:03.650497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:03.668 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:03.663677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:13.924 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:13.919632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:13.937 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:13.931107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:24.168 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:24.164224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:24.182 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:24.178654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:34.434 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:34.430459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:34.449 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:34.445670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:44.684 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:44.680322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:44.698 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:44.694576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:54.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:54.937757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:18:54.958 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:18:54.954341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:05.227 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:05.222888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:05.237 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:05.234082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:15.490 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:15.486080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:15.524 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:15.520702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:25.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:25.785461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:25.801 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:25.798249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:36.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:36.022504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:36.040 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:36.036939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:46.301 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:46.296914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:46.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:46.310165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:56.534 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:56.529784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:19:56.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:19:56.542374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:06.770 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:06.765998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:06.786 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:06.782128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:17.006 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:17.001884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:17.023 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:17.018833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:27.248 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:27.244266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:27.261 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:27.257948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:37.483 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:37.479736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:37.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:37.543790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:47.768 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:47.764553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:47.783 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:47.779423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:58.022 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:58.018965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:20:58.040 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:20:58.036727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:08.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:08.309862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:08.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:08.322531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:18.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:18.550490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:18.579 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:18.574631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:20.563 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:20.559978 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T20:21:21.209 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:21.205407 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:21:21.194+0000 7f21228fe340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:21:21.211 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:21:21.194+0000 7f21228fe340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:21:21.211 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:21:21.194+0000 7f21228fe340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:21:21.211 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:21:21.212 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:21.205456 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:21:21.194+0000 7f21228fe340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:21:21.212 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:21:21.194+0000 7f21228fe340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:21:21.213 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:21:21.194+0000 7f21228fe340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:21:21.213 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:21:28.801 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:28.797521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:28.834 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:28.830798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:39.070 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:39.066618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:39.081 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:39.078193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:49.298 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:49.294267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:49.307 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:49.304426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:59.534 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:59.530683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:21:59.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:21:59.545124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:09.800 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:09.796463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:09.809 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:09.806361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:20.050 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:20.046861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:20.067 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:20.063374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:30.300 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:30.296503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:30.318 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:30.314664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:40.537 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:40.533788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:40.553 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:40.549996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:50.778 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:50.774390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:22:50.792 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:22:50.789302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:01.020 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:01.016821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:01.040 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:01.034607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:11.260 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:11.256632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:11.272 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:11.269521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:21.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:21.492007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:21.542 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:21.539183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:31.791 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:31.787370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:31.805 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:31.801781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:42.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:42.025133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:42.040 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:42.037000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:52.266 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:52.263412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:23:52.286 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:23:52.280201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:02.506 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:02.502817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:02.519 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:02.516491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:12.746 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:12.743236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:12.761 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:12.758190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:22.986 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:22.983088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:23.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:22.996767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:33.218 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:33.215248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:33.238 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:33.235422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:43.462 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:43.458445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:43.491 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:43.488059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:53.712 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:53.708871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:24:53.725 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:24:53.722008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:03.967 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:03.963654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:03.981 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:03.978109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:14.204 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:14.201589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:14.220 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:14.217085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:24.439 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:24.436021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:24.495 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:24.492530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:34.717 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:34.714216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:34.734 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:34.731334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:44.999 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:44.995571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:45.010 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:45.007728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:55.245 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:55.242197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:25:55.256 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:25:55.253870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:05.474 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:05.470837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:05.490 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:05.487127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:15.709 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:15.705676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:15.866 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:15.722383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:25.953 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:25.950120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:25.968 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:25.965002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:36.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:36.195078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:36.213 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:36.210106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:46.437 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:46.434242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:46.451 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:46.447364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:56.668 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:56.665402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:26:56.683 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:26:56.680431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:06.910 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:06.907167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:06.974 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:06.920636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:17.149 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:17.145979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:17.164 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:17.161476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:27.391 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:27.388158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:27.405 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:27.402276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:37.652 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:37.649125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:37.669 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:37.666704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:47.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:47.888900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:47.904 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:47.901807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:58.125 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:58.122125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:27:58.368 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:27:58.133056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:08.357 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:08.353832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:08.374 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:08.370123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:18.603 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:18.600524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:18.622 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:18.616074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:28.843 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:28.840836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:28.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:28.854904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:39.109 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:39.106127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:39.121 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:39.118573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:49.346 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:49.343179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:49.393 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:49.360804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:59.583 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:59.580114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:28:59.601 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:28:59.598162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:00.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:00.022627 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 20:28:59.988094008 +0000 UTC m=+10360.297438110 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T20:29:09.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:09.822161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:09.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:09.836413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:20.067 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:20.064506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:20.122 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:20.120100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:30.350 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:30.346856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:30.364 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:30.361781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:40.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:40.585242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:40.627 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:40.624998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:50.849 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:50.846199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:29:50.920 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:29:50.916676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:00.992 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:00.989156 E | ceph-spec: failed to update cluster condition to {Type:Deleting Status:True Reason:ClusterDeleting Message:Deleting the CephCluster LastHeartbeatTime:2022-03-07 20:30:00.917834648 +0000 UTC m=+10421.227178715 LastTransitionTime:2022-03-07 18:01:54 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T20:30:01.194 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:01.191893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:01.224 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:01.221590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:11.446 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:11.443282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:11.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:11.460143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:21.687 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:21.683656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:21.702 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:21.698626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:31.925 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:31.922651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:31.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:31.936220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:42.161 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:42.158879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:42.177 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:42.174840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:52.405 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:52.402793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:30:52.429 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:30:52.426959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:02.656 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:02.653291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:02.707 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:02.704403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:12.925 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:12.921988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:12.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:12.935001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:23.178 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:23.175084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:23.192 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:23.189655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:33.416 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:33.413502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:33.429 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:33.426436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:43.652 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:43.649064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:43.700 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:43.697710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:53.927 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:53.924941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:31:53.942 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:31:53.940073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:04.164 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:04.161468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:04.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:04.177358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:14.397 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:14.394540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:14.415 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:14.412934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:24.639 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:24.637002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:24.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:24.652234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:34.881 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:34.878211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:34.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:34.891804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:45.150 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:45.147792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:45.165 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:45.163050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:55.387 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:55.384036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:32:55.398 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:32:55.396128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:05.643 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:05.640138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:05.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:05.651836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:15.876 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:15.873176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:15.891 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:15.888946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:26.112 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:26.109137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:26.125 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:26.123150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:36.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:36.344946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:36.359 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:36.357339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:46.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:46.612422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:46.626 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:46.624032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:56.851 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:56.848939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:33:56.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:33:56.866605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:07.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:07.097874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:07.116 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:07.113195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:17.342 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:17.339516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:17.358 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:17.355528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:27.580 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:27.577992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:27.596 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:27.593879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:37.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:37.814829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:37.832 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:37.829631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:48.056 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:48.053225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:48.067 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:48.065176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:58.294 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:58.291329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:34:58.311 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:34:58.308836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:08.537 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:08.534473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:08.560 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:08.557421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:18.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:18.786283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:18.803 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:18.801085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:29.024 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:29.021635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:29.039 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:29.036955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:39.266 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:39.263463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:39.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:39.277306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:49.502 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:49.499363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:49.514 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:49.511566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:59.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:59.740707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:35:59.758 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:35:59.755874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:09.984 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:09.981104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:09.996 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:09.993896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:20.223 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:20.220354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:20.236 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:20.234222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:30.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:30.460181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:30.523 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:30.520138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:40.779 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:40.776779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:40.793 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:40.790434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:51.016 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:51.013258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:36:51.030 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:36:51.027817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:01.273 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:01.270885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:01.290 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:01.286309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:11.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:11.514500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:11.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:11.527460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:21.753 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:21.750500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:21.770 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:21.767433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:31.991 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:31.988794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:32.005 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:32.002445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:42.226 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:42.223376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:42.239 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:42.236913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:52.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:52.460081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:37:52.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:37:52.475146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:01.219 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:01.216376 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T20:38:01.855 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:01.852319 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:38:01.839+0000 7f31d39d4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:38:01.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:38:01.839+0000 7f31d39d4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:38:01.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:38:01.839+0000 7f31d39d4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:38:01.856 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:38:01.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:01.852370 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:38:01.839+0000 7f31d39d4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:38:01.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:38:01.839+0000 7f31d39d4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:38:01.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:38:01.839+0000 7f31d39d4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:38:01.857 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:38:02.738 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:02.735801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:02.747 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:02.744768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:12.970 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:12.967757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:12.984 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:12.981503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:23.208 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:23.205723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:23.222 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:23.219661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:33.448 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:33.445227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:33.463 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:33.461054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:43.695 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:43.692509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:43.711 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:43.709468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:53.939 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:53.936535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:38:53.957 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:38:53.953114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:04.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:04.177502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:04.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:04.193011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:14.424 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:14.420770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:14.438 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:14.435465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:24.660 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:24.658130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:24.676 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:24.672788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:34.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:34.896981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:34.915 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:34.912683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:45.143 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:45.140164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:45.158 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:45.154791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:55.380 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:55.376887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:39:55.396 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:39:55.393639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:05.617 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:05.615026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:05.635 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:05.632439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:15.862 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:15.859221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:15.877 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:15.874361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:26.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:26.098176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:26.117 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:26.114042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:36.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:36.336355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:36.353 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:36.350503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:46.577 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:46.574322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:46.594 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:46.591686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:56.828 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:56.825976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:40:56.845 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:40:56.842951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:07.177 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:07.174783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:07.188 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:07.185803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:17.448 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:17.446012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:17.462 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:17.459744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:27.679 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:27.676948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:27.697 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:27.694585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:37.922 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:37.919196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:37.934 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:37.931909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:48.162 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:48.158236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:48.178 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:48.174629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:58.427 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:58.424786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:41:58.443 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:41:58.437957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:08.682 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:08.679895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:08.698 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:08.695564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:18.923 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:18.921159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:18.938 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:18.935735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:29.164 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:29.161183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:29.179 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:29.177075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:39.403 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:39.400355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:39.421 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:39.418750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:49.648 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:49.645713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:49.665 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:49.662994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:59.918 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:59.915073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:42:59.942 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:42:59.940107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:10.166 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:10.164080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:10.183 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:10.181618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:20.441 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:20.433594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:20.449 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:20.446974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:30.674 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:30.671179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:30.691 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:30.688601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:40.919 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:40.917082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:40.933 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:40.931422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:51.156 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:51.153868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:43:51.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:43:51.196855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:01.461 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:01.458777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:01.476 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:01.474165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:11.729 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:11.726252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:11.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:11.740781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:21.962 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:21.959990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:21.979 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:21.977057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:32.198 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:32.195921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:32.214 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:32.211723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:42.439 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:42.436568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:42.453 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:42.450561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:52.672 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:52.670303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:44:52.686 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:44:52.684421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:02.906 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:02.904199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:02.947 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:02.945172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:13.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:13.177666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:13.195 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:13.192534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:23.412 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:23.409938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:23.424 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:23.422486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:33.641 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:33.638244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:33.657 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:33.654642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:43.917 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:43.914808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:43.928 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:43.925870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:54.151 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:54.148979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:45:54.167 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:45:54.163734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:04.394 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:04.391888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:04.407 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:04.404852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:14.630 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:14.628069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:14.644 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:14.642430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:24.863 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:24.860525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:24.878 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:24.875696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:35.102 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:35.099589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:35.120 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:35.117263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:45.350 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:45.346975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:45.365 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:45.363107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:55.590 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:55.586922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:46:55.605 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:46:55.602822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:05.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:05.832559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:05.850 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:05.847937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:16.078 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:16.075127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:16.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:16.089117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:16.231 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:16.228498 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 20:47:16.180600562 +0000 UTC m=+11456.489944679 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T20:47:26.371 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:26.368814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:26.385 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:26.383412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:36.603 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:36.600842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:36.620 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:36.617769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:46.842 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:46.839636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:46.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:46.853915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:57.081 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:57.078907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:47:57.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:47:57.095815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:07.362 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:07.359367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:07.373 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:07.371030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:17.593 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:17.590931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:17.609 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:17.606638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:27.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:27.830729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:27.850 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:27.845252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:38.067 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:38.065311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:38.084 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:38.081758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:48.306 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:48.303658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:48.321 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:48.319061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:58.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:58.551363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:48:58.566 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:48:58.562524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:08.788 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:08.783746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:08.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:08.819985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:19.059 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:19.056566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:19.071 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:19.069482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:29.295 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:29.292365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:29.309 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:29.306587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:39.591 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:39.588996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:39.612 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:39.608517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:49.835 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:49.830272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:49:49.861 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:49:49.858492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:00.077 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:00.074945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:00.091 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:00.088944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:10.320 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:10.315120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:10.331 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:10.329335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:20.554 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:20.551581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:20.582 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:20.578705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:30.831 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:30.829053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:30.849 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:30.846728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:41.072 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:41.069739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:41.087 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:41.084971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:51.317 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:51.314547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:50:51.328 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:50:51.325877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:01.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:01.582923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:01.599 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:01.596352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:11.828 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:11.825941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:11.841 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:11.838894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:22.062 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:22.059540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:22.101 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:22.098235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:32.342 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:32.330400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:32.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:32.397113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:42.618 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:42.615538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:42.633 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:42.630286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:52.906 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:52.903434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:51:52.919 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:51:52.916483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:03.150 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:03.147244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:03.225 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:03.217427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:13.473 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:13.470258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:13.488 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:13.485332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:23.713 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:23.710725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:23.729 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:23.726214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:33.951 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:33.948868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:33.968 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:33.965340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:44.188 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:44.185727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:44.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:44.200780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:54.434 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:54.431727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:52:54.445 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:52:54.442714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:04.675 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:04.671747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:04.712 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:04.709548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:14.930 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:14.927788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:14.948 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:14.945545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:25.205 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:25.202187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:25.216 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:25.213636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:35.466 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:35.464186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:35.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:35.476100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:45.701 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:45.698286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:45.716 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:45.714088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:55.941 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:55.938381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:53:55.958 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:53:55.955421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:06.180 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:06.178195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:06.191 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:06.189206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:16.417 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:16.413927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:16.431 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:16.428588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:26.690 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:26.687661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:26.701 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:26.698870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:36.922 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:36.919578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:36.933 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:36.930571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:41.862 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:41.859671 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T20:54:42.628 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:42.624469 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:54:42.611+0000 7f2326d7e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:54:42.629 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:54:42.611+0000 7f2326d7e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:54:42.630 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:54:42.611+0000 7f2326d7e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:54:42.630 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:54:42.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:42.624497 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T20:54:42.611+0000 7f2326d7e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T20:54:42.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:54:42.611+0000 7f2326d7e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T20:54:42.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T20:54:42.611+0000 7f2326d7e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T20:54:42.632 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T20:54:47.161 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:47.158109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:47.174 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:47.171704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:57.397 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:57.394794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:54:57.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:54:57.410833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:07.644 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:07.641755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:07.662 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:07.659584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:17.889 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:17.886174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:17.907 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:17.905153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:28.130 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:28.127293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:28.148 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:28.146019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:38.377 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:38.374406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:38.388 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:38.385914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:48.644 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:48.641257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:48.656 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:48.654407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:58.878 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:58.875823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:55:58.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:55:58.892432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:09.121 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:09.117981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:09.188 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:09.174746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:19.399 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:19.396109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:19.413 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:19.410573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:29.636 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:29.632918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:29.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:29.650441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:39.872 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:39.869308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:39.917 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:39.914838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:50.153 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:50.149467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:56:50.166 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:56:50.160394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:00.383 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:00.379981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:00.397 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:00.394740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:10.617 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:10.614585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:10.631 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:10.629145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:20.877 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:20.874155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:20.894 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:20.891670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:31.113 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:31.110185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:31.128 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:31.125215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:41.353 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:41.349911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:41.366 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:41.363840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:51.622 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:51.619859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:57:51.633 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:57:51.630739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:01.885 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:01.882185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:01.899 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:01.897282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:12.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:12.129535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:12.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:12.144157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:22.371 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:22.369062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:22.382 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:22.379761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:32.601 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:32.598830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:32.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:32.612247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:42.841 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:42.838944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:42.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:42.854261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:53.081 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:53.078921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:58:53.097 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:58:53.094707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:03.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:03.316880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:03.335 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:03.332644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:13.555 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:13.552739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:13.570 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:13.568423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:23.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:23.786697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:23.804 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:23.801982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:34.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:34.026410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:34.044 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:34.041938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:44.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:44.264171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:44.281 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:44.278438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:54.509 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:54.506423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T20:59:54.519 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 20:59:54.517126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:04.743 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:04.740633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:04.757 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:04.755231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:14.979 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:14.976249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:14.994 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:14.992332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:25.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:25.212459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:25.229 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:25.227232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:35.455 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:35.452650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:35.470 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:35.467628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:45.691 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:45.688807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:45.709 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:45.706163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:55.929 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:55.925957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:00:55.942 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:00:55.940360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:06.163 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:06.160585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:06.177 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:06.175477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:16.433 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:16.431203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:16.444 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:16.441726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:26.690 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:26.687152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:26.704 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:26.701708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:36.930 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:36.921939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:36.947 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:36.939587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:47.165 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:47.162889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:47.181 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:47.177967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:57.401 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:57.398409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:01:57.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:01:57.411827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:07.636 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:07.633453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:07.652 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:07.648822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:17.874 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:17.871064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:17.889 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:17.886729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:17.889 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:17.886780 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:28.113 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:28.110453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:28.127 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:28.125121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:38.354 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:38.351103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:38.370 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:38.367845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:48.623 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:48.620737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:48.674 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:48.669531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:58.937 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:58.934517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:02:58.952 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:02:58.950284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:09.173 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:09.170893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:09.186 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:09.184364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:19.407 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:19.404768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:19.421 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:19.418820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:29.643 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:29.640826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:29.662 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:29.660037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:39.883 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:39.880461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:39.898 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:39.895409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:50.122 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:50.119724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:03:50.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:03:50.132583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:00.353 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:00.350337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:00.366 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:00.363891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:10.589 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:10.585871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:10.603 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:10.601150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:20.825 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:20.822591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:20.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:20.836888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:31.063 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:31.060449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:31.081 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:31.079695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:41.308 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:41.305900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:41.320 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:41.317696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:51.572 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:51.569762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:04:51.586 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:04:51.584275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:01.847 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:01.844572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:01.866 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:01.863482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:12.089 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:12.086768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:12.103 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:12.100765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:22.326 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:22.323473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:22.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:22.337122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:32.575 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:32.572647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:32.584 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:32.582398 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 21:05:32.554166906 +0000 UTC m=+12552.863510990 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T21:05:32.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:32.584804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:42.815 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:42.812721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:42.853 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:42.850416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:53.079 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:53.076400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:05:53.092 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:05:53.090374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:03.320 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:03.317143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:03.334 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:03.331442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:13.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:13.554515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:13.570 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:13.567760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:23.791 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:23.788281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:23.805 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:23.802840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:34.076 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:34.074070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:34.086 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:34.084214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:44.309 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:44.306863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:44.325 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:44.322347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:54.548 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:54.545527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:06:54.563 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:06:54.560753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:04.822 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:04.819503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:04.833 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:04.831047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:15.060 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:15.058090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:15.119 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:15.116301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:25.375 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:25.372294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:25.393 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:25.390768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:35.616 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:35.613467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:35.637 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:35.634929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:45.892 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:45.889531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:45.946 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:45.944168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:56.215 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:56.212781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:07:56.232 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:07:56.229875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:06.478 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:06.475052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:06.493 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:06.490618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:16.762 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:16.760238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:16.777 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:16.774858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:27.000 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:26.996660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:27.015 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:27.013009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:37.254 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:37.251628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:37.313 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:37.310729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:47.540 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:47.537589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:47.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:47.553196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:57.802 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:57.799537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:08:57.817 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:08:57.815423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:08.076 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:08.073151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:08.085 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:08.082681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:18.316 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:18.313227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:18.331 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:18.328979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:28.553 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:28.550603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:28.567 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:28.565505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:38.786 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:38.783556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:38.797 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:38.795387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:49.024 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:49.021535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:49.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:49.077111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:59.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:59.302955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:09:59.320 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:09:59.317886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:09.544 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:09.541377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:09.558 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:09.555748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:19.784 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:19.781407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:19.796 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:19.793436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:30.014 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:30.011566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:30.039 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:30.036948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:40.264 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:40.261124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:40.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:40.276426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:50.500 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:50.497008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:10:50.517 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:10:50.512253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:00.738 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:00.735876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:00.752 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:00.749361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:10.974 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:10.971622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:10.993 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:10.991407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:21.222 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:21.219209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:21.236 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:21.234038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:22.633 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:22.629939 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T21:11:23.338 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:23.334868 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T21:11:23.324+0000 7fee4b947340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T21:11:23.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:11:23.324+0000 7fee4b947340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T21:11:23.339 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:11:23.324+0000 7fee4b947340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T21:11:23.340 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T21:11:23.340 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:23.334903 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T21:11:23.324+0000 7fee4b947340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T21:11:23.340 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:11:23.324+0000 7fee4b947340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T21:11:23.340 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:11:23.324+0000 7fee4b947340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T21:11:23.340 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T21:11:31.460 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:31.457320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:31.471 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:31.469001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:41.727 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:41.724089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:41.739 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:41.736123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:51.964 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:51.961131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:11:51.979 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:11:51.976547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:02.200 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:02.197883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:02.217 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:02.214421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:12.440 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:12.438006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:12.455 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:12.452193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:22.678 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:22.675303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:22.693 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:22.689848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:32.918 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:32.915006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:32.932 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:32.929551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:43.176 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:43.173190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:43.192 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:43.189463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:53.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:53.411475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:12:53.433 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:12:53.427796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:03.674 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:03.671045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:03.687 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:03.685363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:13.905 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:13.902993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:13.923 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:13.921015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:24.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:24.144392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:24.163 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:24.160585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:34.397 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:34.393789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:34.414 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:34.412029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:44.637 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:44.634415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:44.651 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:44.648788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:54.881 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:54.878443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:13:54.899 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:13:54.891073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:05.150 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:05.147091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:05.169 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:05.166256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:15.396 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:15.393409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:15.411 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:15.409393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:25.634 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:25.631746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:25.649 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:25.646401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:35.869 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:35.866108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:35.885 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:35.882891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:46.105 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:46.102697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:46.120 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:46.117039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:56.346 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:56.343034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:14:56.360 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:14:56.357812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:06.583 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:06.580135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:06.596 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:06.594181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:16.827 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:16.824329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:16.843 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:16.841306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:27.067 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:27.064549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:27.082 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:27.079488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:37.303 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:37.300422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:37.317 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:37.314272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:47.543 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:47.539998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:47.559 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:47.556597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:57.783 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:57.780829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:15:57.797 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:15:57.794941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:08.020 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:08.017435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:08.037 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:08.034372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:18.267 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:18.264680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:18.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:18.276965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:28.516 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:28.513054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:28.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:28.527234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:38.759 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:38.756431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:38.774 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:38.770380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:49.076 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:49.073860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:49.090 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:49.087836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:59.312 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:59.309397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:16:59.330 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:16:59.325362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:09.549 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:09.546225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:09.568 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:09.565852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:19.789 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:19.786118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:19.804 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:19.801327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:30.026 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:30.023910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:30.042 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:30.039719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:40.271 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:40.268822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:40.287 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:40.285033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:50.510 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:50.507519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:17:50.525 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:17:50.522846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:00.787 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:00.784987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:00.804 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:00.801793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:11.033 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:11.030613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:11.046 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:11.043428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:21.291 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:21.288636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:21.310 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:21.306382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:31.532 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:31.529836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:31.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:31.543322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:41.770 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:41.767081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:41.785 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:41.783121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:52.010 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:52.007450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:18:52.025 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:18:52.022267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:02.248 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:02.245635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:02.277 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:02.264323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:12.489 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:12.486007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:12.550 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:12.543303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:22.763 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:22.760489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:22.776 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:22.773897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:32.999 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:32.992522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:33.008 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:33.005580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:43.227 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:43.223775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:43.242 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:43.238768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:53.469 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:53.466468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:19:53.481 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:19:53.479105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:03.705 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:03.701903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:03.720 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:03.716819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:13.980 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:13.978154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:14.018 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:14.015710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:24.269 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:24.265974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:24.283 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:24.280959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:34.509 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:34.505857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:34.524 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:34.521458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:44.739 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:44.737094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:44.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:44.799517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:55.056 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:55.053030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:20:55.073 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:20:55.069785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:05.298 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:05.295523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:05.312 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:05.310123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:15.531 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:15.528332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:15.546 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:15.544007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:25.762 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:25.759677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:25.775 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:25.772511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:35.998 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:35.995777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:36.013 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:36.010416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:46.296 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:46.292859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:46.308 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:46.306294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:56.536 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:56.533854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:21:56.559 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:21:56.556518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:06.797 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:06.794258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:06.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:06.811397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:17.047 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:17.044894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:17.058 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:17.056103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:27.376 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:27.373227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:27.395 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:27.392001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:37.616 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:37.612961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:37.632 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:37.629157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:47.745 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:47.742225 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-07 21:22:47.731476562 +0000 UTC m=+13588.040820724 LastTransitionTime:2022-03-07 17:40:26 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T21:22:47.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:47.853171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:47.867 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:47.865295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:58.089 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:58.085807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:22:58.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:22:58.128854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:08.354 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:08.351593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:08.367 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:08.364502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:18.592 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:18.589202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:18.609 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:18.606593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:28.831 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:28.827920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:28.887 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:28.883648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:39.116 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:39.113242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:39.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:39.129281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:49.353 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:49.350822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:49.369 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:49.366930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:59.595 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:59.592096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:23:59.614 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:23:59.609637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:09.853 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:09.850153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:09.870 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:09.867783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:20.098 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:20.095853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:20.140 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:20.137419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:30.363 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:30.359546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:30.381 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:30.378464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:40.638 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:40.635438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:40.657 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:40.654248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:50.878 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:50.875368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:24:50.894 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:24:50.892108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:01.118 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:01.115645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:01.131 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:01.129399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:11.356 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:11.353940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:11.372 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:11.369931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:21.593 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:21.590294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:21.609 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:21.605906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:31.832 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:31.829250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:31.849 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:31.846445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:42.075 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:42.072182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:42.088 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:42.086158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:52.314 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:52.310483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:25:52.328 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:25:52.325604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:02.545 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:02.543002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:02.557 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:02.554573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:12.787 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:12.778242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:12.791 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:12.787977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:23.013 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:23.009995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:23.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:23.026544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:33.261 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:33.258200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:33.278 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:33.275343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:43.512 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:43.509746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:43.525 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:43.522208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:53.792 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:53.788895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:26:53.857 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:26:53.854088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:04.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:04.129213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:04.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:04.145160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:14.367 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:14.364656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:14.385 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:14.381957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:24.610 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:24.607592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:24.626 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:24.623490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:34.846 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:34.843578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:34.904 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:34.901786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:45.145 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:45.142150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:45.163 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:45.160423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:55.409 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:55.406924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:27:55.424 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:27:55.422046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:03.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:03.344585 I | op-mon: parsing mon endpoints: c=10.108.168.68:6789,a=10.102.1.250:6789,b=10.103.120.199:6789 2022-03-07T21:28:04.027 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:04.023483 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T21:28:04.012+0000 7f0f9dd2c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T21:28:04.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:28:04.012+0000 7f0f9dd2c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T21:28:04.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:28:04.012+0000 7f0f9dd2c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T21:28:04.029 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T21:28:04.029 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:04.023510 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T21:28:04.012+0000 7f0f9dd2c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T21:28:04.030 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:28:04.012+0000 7f0f9dd2c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T21:28:04.030 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07T21:28:04.012+0000 7f0f9dd2c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T21:28:04.031 INFO:tasks.rook.operator.smithi057.stdout:couldn't init storage provider. : exit status 5 2022-03-07T21:28:05.658 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:05.655589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:05.671 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:05.668749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:15.895 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:15.892244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:15.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:15.909133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:26.132 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:26.129430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:26.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:26.144793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:36.371 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:36.368001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:36.385 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:36.382173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:46.609 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:46.606432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:46.623 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:46.620484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:56.839 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:56.836401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:28:56.854 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:28:56.851598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:07.082 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:07.079237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:07.097 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:07.094321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:17.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:17.344296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:17.363 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:17.360366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:27.587 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:27.584027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:27.602 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:27.599216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:37.828 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:37.825342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:37.846 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:37.843992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:48.071 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:48.067889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:48.090 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:48.087719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:58.316 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:58.313089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:29:58.338 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:29:58.335386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:08.562 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:08.559685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:08.620 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:08.617917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:18.842 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:18.839546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:18.856 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:18.853290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:29.080 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:29.077660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:29.141 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:29.138104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:39.389 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:39.386102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:39.406 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:39.403130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:49.650 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:49.647134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:49.665 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:49.660052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:59.886 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:59.883381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:30:59.901 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:30:59.897968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:10.127 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:10.123890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:10.142 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:10.139061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:20.363 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:20.359985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:20.376 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:20.374003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:30.602 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:30.598812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:30.654 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:30.651962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:40.900 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:40.897145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:40.911 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:40.908518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:51.133 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:51.129953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:31:51.199 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:31:51.195745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:01.420 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:01.416852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:01.435 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:01.432872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:11.694 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:11.691178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:11.711 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:11.708167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:21.937 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:21.933874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:21.949 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:21.947096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:32.172 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:32.169074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:32.187 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:32.184225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:42.408 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:42.404508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:42.422 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:42.419862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:52.655 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:52.652514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:32:52.671 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:32:52.668196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:02.897 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:02.894306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:02.913 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:02.908504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:13.134 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:13.131435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:13.147 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:13.145065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:23.375 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:23.371593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:23.408 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:23.386903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:33.615 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:33.611966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:33.628 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:33.625534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:43.863 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:43.860549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:43.875 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:43.872712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:54.100 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:54.097215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:33:54.113 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:33:54.110846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:04.347 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:04.344087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:04.363 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:04.359894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:14.588 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:14.585396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:14.604 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:14.601359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:24.829 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:24.825006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:24.842 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:24.839559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:35.066 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:35.063023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:35.077 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:35.074698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:45.305 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:45.302255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:45.319 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:45.316124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:55.553 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:55.550225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:34:55.570 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:34:55.567876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:05.798 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:05.794620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:05.814 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:05.811406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:16.039 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:16.036361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:16.052 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:16.049935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:26.279 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:26.276278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:26.296 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:26.292811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:36.516 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:36.513118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:36.530 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:36.527339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:46.762 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:46.759236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:46.774 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:46.771806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:57.011 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:57.007942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:35:57.028 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:35:57.025361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:36:07.255 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:36:07.249066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:36:07.263 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:36:07.260463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:36:17.487 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:36:17.484078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:36:17.501 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:36:17.498480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:36:27.725 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:36:27.721981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T21:36:27.740 INFO:tasks.rook.operator.smithi057.stdout:2022-03-07 21:36:27.737472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-07T23:56:31.388 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-07T23:56:31.407 DEBUG:teuthology.task.console_log:Killing console logger for smithi057 2022-03-07T23:56:31.408 DEBUG:teuthology.task.console_log:Killing console logger for smithi066 2022-03-07T23:56:31.409 DEBUG:teuthology.task.console_log:Killing console logger for smithi085 2022-03-07T23:56:31.409 DEBUG:teuthology.exit:Finished running handlers