2022-03-08T22:44:32.608 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T22:44:32.610 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727284 --owner scheduled_yuriw@teuthology --name yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} --verbose -- /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727284/orig.config.yaml 2022-03-08T22:44:32.676 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T22:44:32.816 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727284 branch: wip-yuri-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} email: null first_in_suite: false job_id: '6727284' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri-testing-2022-03-07-0958 ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 spec: mon: allowMultiplePerNode: false network: provider: host ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 kernel: hwe: true kubeadm: version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-yuri-testing-2022-03-07-0958 sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 targets: smithi176.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD2IraKpzLOoJJkZkLmTahVG7/VYGNRXV1dgeDDJDUOJPlqmiZ4v/lBhpJFTtrAlRwodGh5yjIFasjD6sicCrQEr2jyHhkC19php1v4lPYS7/QbngGvQPpuHzsllDa/3LnQl7ljOhIWGGBBmX1JAbdWsnSvfYlprlFZWUPEbqpuNHgX2PoIbkB69KSbN742xZFbsAygf/9qVSP9dDPHgS2u298YECUhxNk986Weo5qDHMWbp2/xOti+WspVStWEXRlq9+zh0fCH2QhTZbTHxDF6DKoSiTSZsG3Ujjti06KAU0UrmfyU3dpg6S2eW/+NKQfkivGLWZUMbpn6/zFiRr0pcBQdKxrBs5F1rkRfUHUEwlaNY/fb3gSwKX/qtleuaCd9AbPPDA/u6BLb0Ri0xagnH45xnDRWqwURTkFM/0QKEVaFgeJaDj5fz2zDWUeaGGLwHgMyGLZJZuYqa68KoMho8bqNMoIVuTdCwFpc4f2If2yAFHxSLQYPSYSJjoBmxqM= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-08_22:35:06 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T22:44:32.817 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa; will attempt to use it 2022-03-08T22:44:32.818 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks 2022-03-08T22:44:32.818 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T22:44:32.822 INFO:teuthology.task.internal:Checking packages... 2022-03-08T22:44:32.845 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '94cb3430b74fb099b8927e3e6ad50dfd969173f4' 2022-03-08T22:44:32.846 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T22:44:32.846 INFO:teuthology.packaging:ref: None 2022-03-08T22:44:32.846 INFO:teuthology.packaging:tag: None 2022-03-08T22:44:32.847 INFO:teuthology.packaging:branch: wip-yuri-testing-2022-03-07-0958 2022-03-08T22:44:32.847 INFO:teuthology.packaging:sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:44:32.848 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri-testing-2022-03-07-0958 2022-03-08T22:44:33.053 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11002-g94cb3430-1focal 2022-03-08T22:44:33.053 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T22:44:33.063 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T22:44:33.064 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T22:44:33.092 INFO:teuthology.task.internal:Saving configuration 2022-03-08T22:44:33.107 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T22:44:33.150 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T22:44:33.175 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:35:06-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6727284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:39:20.927955', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDb3H5pMylQrsBd41biJZbTHpG3l7PONmTcfXq2IZ8nN'} 2022-03-08T22:44:33.176 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T22:44:33.195 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T22:44:33.197 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T22:44:33.251 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f202130c9d8>, signals=[15]) 2022-03-08T22:44:33.252 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T22:44:33.262 INFO:teuthology.task.internal:Opening connections... 2022-03-08T22:44:33.263 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2022-03-08T22:44:33.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:44:33.317 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T22:44:33.326 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2022-03-08T22:44:33.441 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2022-03-08T22:44:33.441 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2022-03-08T22:44:33.447 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Ubuntu" 2022-03-08T22:44:33.447 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:44:33.448 INFO:teuthology.orchestra.run.smithi176.stdout:ID=ubuntu 2022-03-08T22:44:33.448 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE=debian 2022-03-08T22:44:33.448 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:44:33.448 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="20.04" 2022-03-08T22:44:33.449 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:44:33.449 INFO:teuthology.orchestra.run.smithi176.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:44:33.449 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:44:33.449 INFO:teuthology.orchestra.run.smithi176.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:44:33.449 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_CODENAME=focal 2022-03-08T22:44:33.450 INFO:teuthology.orchestra.run.smithi176.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:44:33.453 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2022-03-08T22:44:33.473 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T22:44:33.485 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T22:44:33.519 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T22:44:33.520 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:44:33.527 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T22:44:33.553 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T22:44:33.554 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:44:33.575 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T22:44:33.612 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T22:44:33.635 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T22:44:33.636 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T22:44:33.636 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-08T22:44:33.636 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T22:44:33.637 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2022-03-08T22:44:33.644 INFO:teuthology.orchestra.run.smithi176.stdout:5.4.0-97-generic 2022-03-08T22:44:33.644 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.4.0-97-generic 2022-03-08T22:44:33.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2022-03-08T22:44:33.735 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2022-03-08T22:44:33.907 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:44:34.065 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:44:34.161 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:44:34.280 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:44:34.456 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:44:34.551 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:44:34.557 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:44:34.558 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:44:34.570 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:44:34.578 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:44:34.579 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:44:34.582 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:44:34.582 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:44:34.597 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:44:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:44:34.607 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:44:34.618 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:44:34.618 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:44:34.619 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:44:34.619 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:44:34.702 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:44:34.998 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:44:35.037 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:44:35.047 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:44:35.047 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:44:35.048 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:44:35.083 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:44:35.086 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:44:35.086 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:44:35.110 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:44:35.129 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:44:35.163 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:44:35.164 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:44:35.164 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:44:35.165 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:44:35.216 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:44:35.216 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:44:35.217 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:44:35.218 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:44:35.219 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:44:35.245 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:44:35.327 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:44:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:44:37.648 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 10.2 MB in 2s (5,582 kB/s) 2022-03-08T22:44:38.825 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:44:38.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:44:38.918 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:44:39.091 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:44:39.091 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:44:39.243 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2022-03-08T22:44:39.243 INFO:teuthology.orchestra.run.smithi176.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:44:39.243 INFO:teuthology.orchestra.run.smithi176.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:44:39.243 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:44:39.244 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2022-03-08T22:44:39.244 INFO:teuthology.orchestra.run.smithi176.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:44:39.244 INFO:teuthology.orchestra.run.smithi176.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:44:39.291 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2022-03-08T22:44:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:44:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:44:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:44:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: thermald upower usbmuxd 2022-03-08T22:44:39.473 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:44:39.473 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:44:39.474 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:44:39.474 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:44:39.742 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:44:39.756 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:44:39.757 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:44:39.844 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:44:39.845 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:44:40.728 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:44:40.993 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:44:42.708 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:44:42.708 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:44:42.709 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:44:42.714 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:44:42.715 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:44:43.111 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 80.1 MB in 3s (23.4 MB/s) 2022-03-08T22:44:43.292 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:44:44.751 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:44:44.756 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:44:44.897 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:44:45.258 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:44:45.272 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:44:45.316 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:44:45.661 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:44:45.673 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:44:45.711 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:44:46.005 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:44:46.019 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:44:46.055 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:44:46.416 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:44:46.431 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:44:46.466 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:44:46.811 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:44:46.826 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:44:46.869 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:49.103 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:44:49.116 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:44:49.216 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:49.730 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:44:49.746 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:44:49.789 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:44:55.595 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:44:55.610 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:44:55.653 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:44:55.972 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:44:55.986 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:44:56.036 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:44:56.422 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package thermald. 2022-03-08T22:44:56.436 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:44:56.480 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:44:57.012 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package upower. 2022-03-08T22:44:57.028 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:44:57.073 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:44:57.836 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:44:57.849 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:44:57.895 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:44:58.190 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:44:58.317 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:44:58.434 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:44:58.560 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:44:58.686 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:44:58.812 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:44:58.947 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:44:59.378 INFO:teuthology.orchestra.run.smithi176.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:44:59.432 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:44:59.613 INFO:teuthology.orchestra.run.smithi176.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:44:59.621 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:44:59.621 INFO:teuthology.orchestra.run.smithi176.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:45:00.098 INFO:teuthology.orchestra.run.smithi176.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:45:00.322 INFO:teuthology.orchestra.run.smithi176.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:45:00.369 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:45:00.667 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:00.667 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:45:01.149 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:02.435 INFO:teuthology.orchestra.run.smithi176.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:45:02.436 INFO:teuthology.orchestra.run.smithi176.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:45:02.592 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:05.228 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:45:05.353 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:45:09.518 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:45:10.170 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:45:10.270 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:45:10.473 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:45:10.474 INFO:teuthology.orchestra.run.smithi176.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:33.948 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:45:33.948 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:45:33.963 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:45:33.985 INFO:teuthology.orchestra.run.smithi176.stdout:Generating grub configuration file ... 2022-03-08T22:45:34.308 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:45:34.325 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:34.677 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:45:34.689 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:45:37.102 INFO:teuthology.orchestra.run.smithi176.stdout:done 2022-03-08T22:45:37.432 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:45:37.463 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:45:37.463 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2022-03-08T22:45:37.463 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2022-03-08T22:45:37.463 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2022-03-08T22:45:37.464 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 19 2022-03-08T22:45:37.464 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:45:37.464 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2022-03-08T22:45:37.464 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:45:37.465 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:45:37.465 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:45:37.465 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:45:37.466 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2022-03-08T22:45:37.466 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2022-03-08T22:45:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:45:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2022-03-08T22:45:37.467 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:45:37.467 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:45:37.467 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:45:37.468 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T22:45:37.468 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:45:37.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2022-03-08T22:45:37.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2022-03-08T22:45:37.797 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:45:37.807 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:45:37.890 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:45:37.972 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:45:39.339 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:45:39.362 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:45:39.428 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:45:39.584 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:45:39.585 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:45:39.756 INFO:teuthology.orchestra.run.smithi176.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:45:39.756 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:45:39.758 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:45:39.774 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:45:39.774 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2022-03-08T22:45:39.775 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2022-03-08T22:45:39.775 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2022-03-08T22:45:39.775 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 19 2022-03-08T22:45:39.775 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:45:39.776 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2022-03-08T22:45:39.776 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:45:39.776 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:45:39.776 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:45:39.776 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:45:39.777 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2022-03-08T22:45:39.777 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2022-03-08T22:45:39.777 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:45:39.777 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2022-03-08T22:45:39.778 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2022-03-08T22:45:39.785 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.JY85LlSXPI 2022-03-08T22:45:39.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub/grub.cfg /tmp/tmp.JY85LlSXPI 2022-03-08T22:45:39.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.JY85LlSXPI 2022-03-08T22:45:39.880 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.JY85LlSXPI is 8KB 2022-03-08T22:45:39.895 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.JY85LlSXPI 2022-03-08T22:45:39.902 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:45:39.957 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:45:39.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:45:39.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:45:39.997 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:45:39.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo update-grub 2022-03-08T22:45:41.190 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:45:41.200 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:45:41.217 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2022-03-08T22:45:41.429 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:45:41.441 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:45:41.822 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:45:41.833 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:45:43.199 INFO:teuthology.orchestra.run.smithi176.stderr:done 2022-03-08T22:45:43.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2022-03-08T22:46:13.226 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T22:46:13.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-03-08T22:46:13.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:46:31.631 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2022-03-08T22:47:01.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-03-08T22:47:01.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:20.011 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2022-03-08T22:47:50.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-03-08T22:47:50.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:47:50.371 DEBUG:teuthology.orchestra.run.smithi176:> true 2022-03-08T22:47:50.865 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-03-08T22:47:50.865 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-08T22:47:50.865 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2022-03-08T22:47:50.915 INFO:teuthology.orchestra.run.smithi176.stdout:5.13.0-35-generic 2022-03-08T22:47:50.915 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:47:50.915 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:47:50.916 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:47:51.916 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T22:47:51.917 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:47:52.251 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2022-03-08T22:47:52.279 DEBUG:teuthology.parallel:result is None 2022-03-08T22:47:52.280 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T22:47:52.289 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T22:47:52.289 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:47:52.296 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T22:47:52.409 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T22:47:52.458 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T22:47:52.459 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:47:52.518 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T22:47:52.528 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T22:47:52.529 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:47:52.549 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:47:52.558 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:47:52.560 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T22:47:52.570 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T22:47:52.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:47:52.617 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T22:47:52.629 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T22:47:52.629 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:47:52.636 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:47:52.643 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:47:52.692 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:47:52.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:47:52.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2022-03-08T22:47:52.761 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T22:47:52.770 INFO:teuthology.task.internal:Starting timer... 2022-03-08T22:47:52.771 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T22:47:52.801 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T22:47:52.851 DEBUG:teuthology.task.selinux:Excluding smithi176: OS 'ubuntu' does not support SELinux 2022-03-08T22:47:52.852 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T22:47:52.852 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T22:47:52.852 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T22:47:52.853 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T22:47:52.900 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T22:47:52.900 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T22:47:52.921 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T22:47:52.922 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T22:50:50.121 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2022-03-08T22:50:50.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-03-08T22:50:50.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:50:50.166 DEBUG:teuthology.orchestra.run.smithi176:> true 2022-03-08T22:50:50.288 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T22:50:50.299 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T22:50:50.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T22:50:50.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T22:50:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T22:50:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Command line: ntpd -gq 2022-03-08T22:50:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: proto: precision = 0.066 usec (-24) 2022-03-08T22:50:50.335 INFO:teuthology.orchestra.run.smithi176.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:50:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T22:50:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:50:50.336 INFO:teuthology.orchestra.run.smithi176.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T22:50:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T22:50:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T22:50:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T22:50:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listen normally on 3 enp3s0f1 172.21.15.176:123 2022-03-08T22:50:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listen normally on 4 lo [::1]:123 2022-03-08T22:50:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f61%5]:123 2022-03-08T22:50:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:50 ntpd[9042]: Listening on routing socket on fd #22 for interface updates 2022-03-08T22:50:51.334 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:51 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:51.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:51 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:51.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:51 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:51.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:51 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:51.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:51 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:51.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:51 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:52.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:52 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:52.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:52 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:52.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:52 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:52.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:52 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:52.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:52 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:52.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:52 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:53.334 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:53 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:53.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:53 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:53.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:53 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:53.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:53 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:53.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:53 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:53.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:53 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:54.334 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:54 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:54.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:54 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:54.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:54 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:54.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:54 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:54.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:54 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:54.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:54 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:55.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:55.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:55.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:55.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:55.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:55.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:55.337 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:55.337 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:55.337 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:55.337 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:55.338 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:55.338 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:55.338 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:55.338 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:55.338 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:55.339 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:55.339 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:55.339 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:55 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:56.334 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:56 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:56.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:56 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:56.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:56 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:56.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:56 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:56.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:56 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:56.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:56 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:57.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:57.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:57.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:57.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:57.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:57.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:57.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T22:50:57.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T22:50:57.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T22:50:57.337 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T22:50:57.337 INFO:teuthology.orchestra.run.smithi176.stdout: 8 Mar 22:50:57 ntpd[9042]: ntpd: time slew -0.000116 s 2022-03-08T22:50:57.337 INFO:teuthology.orchestra.run.smithi176.stdout:ntpd: time slew -0.000116s 2022-03-08T22:50:57.338 INFO:teuthology.orchestra.run.smithi176.stderr: 8 Mar 22:50:57 ntpd[9042]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T22:50:57.401 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T22:50:57.401 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2022-03-08T22:50:57.402 INFO:teuthology.orchestra.run.smithi176.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:57.402 INFO:teuthology.orchestra.run.smithi176.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:57.402 INFO:teuthology.orchestra.run.smithi176.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:57.402 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T22:50:57.403 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T22:50:57.420 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T22:50:57.420 INFO:tasks.kubeadm:Config: {'version': '1.21'} 2022-03-08T22:50:57.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T22:50:57.474 INFO:teuthology.orchestra.run.smithi176.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T22:50:57.482 INFO:teuthology.orchestra.run.smithi176.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T22:50:57.492 INFO:teuthology.orchestra.run.smithi176.stdout:net.ipv4.ip_forward = 1 2022-03-08T22:50:57.550 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:50:57.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T22:50:57.564 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:50:57.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/docker/daemon.json 2022-03-08T22:50:57.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl restart docker || true 2022-03-08T22:50:58.450 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T22:50:58.457 INFO:tasks.kubeadm:172.21.15.176 is in 172.21.0.0/20 at pos 4015 2022-03-08T22:50:58.470 INFO:tasks.kubeadm:Pod subnet: 10.253.120.0/21 2022-03-08T22:50:58.470 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T22:50:58.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T22:50:58.491 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:50:58.492 INFO:teuthology.orchestra.run.smithi176.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:50:58.492 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:50:58.605 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:50:58.716 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:50:58.798 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:50:58.882 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:00.077 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:51:00.245 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:51:00.246 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:51:00.336 INFO:teuthology.orchestra.run.smithi176.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:00.347 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:51:00.347 INFO:teuthology.orchestra.run.smithi176.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:00.347 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:51:00.400 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:51:00.560 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:51:00.560 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:51:00.721 INFO:teuthology.orchestra.run.smithi176.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T22:51:00.721 INFO:teuthology.orchestra.run.smithi176.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T22:51:00.721 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2022-03-08T22:51:00.722 INFO:teuthology.orchestra.run.smithi176.stdout: apt-transport-https 2022-03-08T22:51:00.905 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:00.906 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 4,680 B of archives. 2022-03-08T22:51:00.906 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T22:51:00.906 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T22:51:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:01.246 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:01.246 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:01.272 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:01.272 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:01.272 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:01.278 INFO:teuthology.orchestra.run.smithi176.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:01.328 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 4,680 B in 0s (26.5 kB/s) 2022-03-08T22:51:01.412 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T22:51:01.456 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T22:51:01.459 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T22:51:01.502 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T22:51:01.775 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T22:51:02.254 INFO:teuthology.orchestra.run.smithi176.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T22:51:02.271 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:51:02.271 INFO:teuthology.orchestra.run.smithi176.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:02.271 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:51:02.524 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:51:02.525 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:51:02.595 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:51:02.677 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:51:02.849 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:51:03.170 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T22:51:03.301 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 63.3 kB in 1s (71.4 kB/s) 2022-03-08T22:51:04.170 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:51:04.346 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:51:04.347 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:51:04.442 INFO:teuthology.orchestra.run.smithi176.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T22:51:04.458 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:51:04.458 INFO:teuthology.orchestra.run.smithi176.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T22:51:04.458 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:51:04.514 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:51:04.691 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:51:04.692 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:51:04.841 INFO:teuthology.orchestra.run.smithi176.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T22:51:04.842 INFO:teuthology.orchestra.run.smithi176.stdout:bridge-utils set to manually installed. 2022-03-08T22:51:04.842 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2022-03-08T22:51:04.842 INFO:teuthology.orchestra.run.smithi176.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T22:51:04.843 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2022-03-08T22:51:04.843 INFO:teuthology.orchestra.run.smithi176.stdout: nftables 2022-03-08T22:51:04.894 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2022-03-08T22:51:04.895 INFO:teuthology.orchestra.run.smithi176.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T22:51:05.088 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:51:05.088 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 73.6 MB of archives. 2022-03-08T22:51:05.088 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T22:51:05.088 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T22:51:05.142 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T22:51:05.279 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T22:51:05.416 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T22:51:05.463 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T22:51:06.046 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T22:51:06.586 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T22:51:06.795 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T22:51:07.198 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T22:51:07.198 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T22:51:07.199 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: falling back to frontend: Readline 2022-03-08T22:51:07.207 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T22:51:07.207 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T22:51:07.207 INFO:teuthology.orchestra.run.smithi176.stderr:debconf: falling back to frontend: Teletype 2022-03-08T22:51:07.212 INFO:teuthology.orchestra.run.smithi176.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T22:51:07.261 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 73.6 MB in 2s (35.4 MB/s) 2022-03-08T22:51:07.351 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package conntrack. 2022-03-08T22:51:07.393 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T22:51:07.397 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T22:51:07.441 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T22:51:07.702 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cri-tools. 2022-03-08T22:51:07.714 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T22:51:07.743 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T22:51:08.755 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ebtables. 2022-03-08T22:51:08.767 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T22:51:08.822 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T22:51:09.082 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T22:51:09.095 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T22:51:09.132 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:10.559 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package socat. 2022-03-08T22:51:10.572 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T22:51:10.609 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T22:51:11.001 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package kubelet. 2022-03-08T22:51:11.014 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T22:51:11.051 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T22:51:13.412 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package kubectl. 2022-03-08T22:51:13.425 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T22:51:13.470 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T22:51:14.658 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package kubeadm. 2022-03-08T22:51:14.672 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T22:51:14.716 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T22:51:15.846 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T22:51:15.944 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T22:51:16.072 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T22:51:16.894 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T22:51:16.995 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T22:51:17.121 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T22:51:17.247 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T22:51:17.461 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T22:51:17.965 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T22:51:18.342 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:51:19.199 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T22:51:32.641 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T22:51:35.426 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T22:51:37.507 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T22:51:40.881 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T22:51:42.298 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T22:51:47.355 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T22:51:49.993 INFO:teuthology.orchestra.run.smithi176.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T22:51:50.000 INFO:tasks.kubeadm:Token: abcdef.rc9m9a514696l1h4 2022-03-08T22:51:50.001 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): '172.21.15.176'} 2022-03-08T22:51:50.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo kubeadm init --node-name smithi176 --token abcdef.rc9m9a514696l1h4 --pod-network-cidr 10.253.120.0/21 2022-03-08T22:51:50.190 INFO:teuthology.orchestra.run.smithi176.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T22:51:50.190 INFO:teuthology.orchestra.run.smithi176.stdout:[preflight] Running pre-flight checks 2022-03-08T22:51:50.585 INFO:teuthology.orchestra.run.smithi176.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T22:51:50.585 INFO:teuthology.orchestra.run.smithi176.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T22:51:50.586 INFO:teuthology.orchestra.run.smithi176.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T22:51:50.898 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T22:51:51.057 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "ca" certificate and key 2022-03-08T22:51:51.306 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T22:51:51.307 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi176] and IPs [10.96.0.1 172.21.15.176] 2022-03-08T22:51:51.705 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T22:51:51.843 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T22:51:52.364 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T22:51:52.608 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T22:51:52.776 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T22:51:52.777 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi176] and IPs [172.21.15.176 127.0.0.1 ::1] 2022-03-08T22:51:52.981 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T22:51:52.981 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi176] and IPs [172.21.15.176 127.0.0.1 ::1] 2022-03-08T22:51:53.448 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T22:51:53.576 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T22:51:53.700 INFO:teuthology.orchestra.run.smithi176.stdout:[certs] Generating "sa" key and public key 2022-03-08T22:51:53.701 INFO:teuthology.orchestra.run.smithi176.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T22:51:53.945 INFO:teuthology.orchestra.run.smithi176.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T22:51:54.072 INFO:teuthology.orchestra.run.smithi176.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T22:51:54.298 INFO:teuthology.orchestra.run.smithi176.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T22:51:54.391 INFO:teuthology.orchestra.run.smithi176.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T22:51:54.402 INFO:teuthology.orchestra.run.smithi176.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T22:51:54.404 INFO:teuthology.orchestra.run.smithi176.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T22:51:54.405 INFO:teuthology.orchestra.run.smithi176.stdout:[kubelet-start] Starting the kubelet 2022-03-08T22:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T22:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T22:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T22:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T22:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T22:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T22:52:10.627 INFO:teuthology.orchestra.run.smithi176.stdout:[apiclient] All control plane components are healthy after 16.007877 seconds 2022-03-08T22:52:10.628 INFO:teuthology.orchestra.run.smithi176.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T22:52:10.644 INFO:teuthology.orchestra.run.smithi176.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T22:52:10.644 INFO:teuthology.orchestra.run.smithi176.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T22:52:11.200 INFO:teuthology.orchestra.run.smithi176.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T22:52:11.200 INFO:teuthology.orchestra.run.smithi176.stdout:[mark-control-plane] Marking the node smithi176 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T22:52:11.200 INFO:teuthology.orchestra.run.smithi176.stdout:[mark-control-plane] Marking the node smithi176 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T22:52:11.717 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] Using token: abcdef.rc9m9a514696l1h4 2022-03-08T22:52:11.717 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T22:52:11.735 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T22:52:11.769 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T22:52:11.777 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T22:52:11.786 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T22:52:11.799 INFO:teuthology.orchestra.run.smithi176.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T22:52:11.876 INFO:teuthology.orchestra.run.smithi176.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T22:52:12.447 INFO:teuthology.orchestra.run.smithi176.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T22:52:12.603 INFO:teuthology.orchestra.run.smithi176.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T22:52:12.604 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.605 INFO:teuthology.orchestra.run.smithi176.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T22:52:12.605 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.605 INFO:teuthology.orchestra.run.smithi176.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T22:52:12.605 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.606 INFO:teuthology.orchestra.run.smithi176.stdout: mkdir -p $HOME/.kube 2022-03-08T22:52:12.606 INFO:teuthology.orchestra.run.smithi176.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T22:52:12.606 INFO:teuthology.orchestra.run.smithi176.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T22:52:12.606 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.607 INFO:teuthology.orchestra.run.smithi176.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T22:52:12.607 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.607 INFO:teuthology.orchestra.run.smithi176.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T22:52:12.607 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.607 INFO:teuthology.orchestra.run.smithi176.stdout:You should now deploy a pod network to the cluster. 2022-03-08T22:52:12.608 INFO:teuthology.orchestra.run.smithi176.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T22:52:12.608 INFO:teuthology.orchestra.run.smithi176.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T22:52:12.608 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.608 INFO:teuthology.orchestra.run.smithi176.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T22:52:12.609 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:52:12.609 INFO:teuthology.orchestra.run.smithi176.stdout:kubeadm join 172.21.15.176:6443 --token abcdef.rc9m9a514696l1h4 \ 2022-03-08T22:52:12.609 INFO:teuthology.orchestra.run.smithi176.stdout: --discovery-token-ca-cert-hash sha256:4bbc1cbf0d37e97515447682c62bd94829d1b79ea6294ee69965b78909e16151 2022-03-08T22:52:12.611 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:12.612 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T22:52:12.622 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T22:52:12.622 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T22:52:12.683 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:12.684 DEBUG:teuthology.orchestra.run.smithi176:> dd of=.kube/config 2022-03-08T22:52:12.689 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:12.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/root/.kube/config 2022-03-08T22:52:12.700 DEBUG:teuthology.orchestra.run.smithi176:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-08T22:52:15.428 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-08T22:52:15.479 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-08T22:52:15.565 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-08T22:52:15.614 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-08T22:52:15.737 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-08T22:52:15.779 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-08T22:52:15.830 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-08T22:52:15.907 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-08T22:52:15.957 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-08T22:52:16.005 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-08T22:52:16.092 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-08T22:52:16.138 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-08T22:52:16.222 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-08T22:52:16.254 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-08T22:52:16.306 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-08T22:52:16.413 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-08T22:52:16.706 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-08T22:52:16.781 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-08T22:52:16.898 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-08T22:52:16.961 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-08T22:52:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-08T22:52:17.080 INFO:teuthology.orchestra.run.smithi176.stdout:namespace/tigera-operator created 2022-03-08T22:52:17.091 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:52:17.095 INFO:teuthology.orchestra.run.smithi176.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-08T22:52:17.104 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/tigera-operator created 2022-03-08T22:52:17.181 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T22:52:17.217 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T22:52:17.245 INFO:teuthology.orchestra.run.smithi176.stdout:deployment.apps/tigera-operator created 2022-03-08T22:52:17.251 DEBUG:teuthology.orchestra.run.smithi176:> kubectl create -f - 2022-03-08T22:52:19.836 INFO:teuthology.orchestra.run.smithi176.stdout:installation.operator.tigera.io/default created 2022-03-08T22:52:19.844 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T22:52:19.844 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:19.845 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:52:19.852 INFO:tasks.kubeadm:Scratch devices on smithi176: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:52:19.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T22:52:19.880 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records in 2022-03-08T22:52:19.880 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2022-03-08T22:52:19.881 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0176782 s, 593 MB/s 2022-03-08T22:52:19.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T22:52:19.960 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records in 2022-03-08T22:52:19.960 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2022-03-08T22:52:19.960 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0177635 s, 590 MB/s 2022-03-08T22:52:19.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T22:52:19.988 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records in 2022-03-08T22:52:19.988 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2022-03-08T22:52:19.989 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150304 s, 698 MB/s 2022-03-08T22:52:19.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T22:52:20.019 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records in 2022-03-08T22:52:20.019 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2022-03-08T22:52:20.019 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155955 s, 672 MB/s 2022-03-08T22:52:20.027 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T22:52:20.027 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi176-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi176 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi176-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi176 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi176-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi176 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi176-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi176 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T22:52:20.028 DEBUG:teuthology.orchestra.run.smithi176:> kubectl create -f - 2022-03-08T22:52:20.966 INFO:teuthology.orchestra.run.smithi176.stdout:persistentvolume/smithi176-lv-1 created 2022-03-08T22:52:20.974 INFO:teuthology.orchestra.run.smithi176.stdout:persistentvolume/smithi176-lv-2 created 2022-03-08T22:52:20.983 INFO:teuthology.orchestra.run.smithi176.stdout:persistentvolume/smithi176-lv-3 created 2022-03-08T22:52:20.991 INFO:teuthology.orchestra.run.smithi176.stdout:persistentvolume/smithi176-lv-4 created 2022-03-08T22:52:21.019 INFO:teuthology.orchestra.run.smithi176.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T22:52:21.024 DEBUG:teuthology.orchestra.run.smithi176:> kubectl taint node smithi176 node-role.kubernetes.io/master- || true 2022-03-08T22:52:21.094 INFO:teuthology.orchestra.run.smithi176.stdout:node/smithi176 untainted 2022-03-08T22:52:21.097 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T22:52:21.097 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T22:52:21.121 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T22:52:21.122 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:21.122 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:52:21.130 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T22:52:21.131 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T22:52:21.181 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T22:52:21.181 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:21.182 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-08T22:52:21.182 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:21.182 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2022-03-08 22:52:19.895143696 +0000 2022-03-08T22:52:21.182 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2022-03-08 22:52:19.895143696 +0000 2022-03-08T22:52:21.182 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2022-03-08 22:52:19.895143696 +0000 2022-03-08T22:52:21.183 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2022-03-08T22:52:21.191 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2022-03-08T22:52:21.191 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2022-03-08T22:52:21.191 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.00059649 s, 858 kB/s 2022-03-08T22:52:21.194 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T22:52:21.200 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T22:52:21.201 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:21.201 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-08T22:52:21.201 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:21.201 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2022-03-08 22:52:19.967143694 +0000 2022-03-08T22:52:21.202 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2022-03-08 22:52:19.963143695 +0000 2022-03-08T22:52:21.202 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2022-03-08 22:52:19.963143695 +0000 2022-03-08T22:52:21.202 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2022-03-08T22:52:21.206 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2022-03-08T22:52:21.207 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2022-03-08T22:52:21.207 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000478804 s, 1.1 MB/s 2022-03-08T22:52:21.210 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T22:52:21.217 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T22:52:21.217 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:21.217 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 510 Links: 1 2022-03-08T22:52:21.218 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:21.218 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2022-03-08 22:52:19.991143694 +0000 2022-03-08T22:52:21.218 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2022-03-08 22:52:19.991143694 +0000 2022-03-08T22:52:21.218 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2022-03-08 22:52:19.991143694 +0000 2022-03-08T22:52:21.219 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2022-03-08T22:52:21.227 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2022-03-08T22:52:21.227 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2022-03-08T22:52:21.227 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000527903 s, 970 kB/s 2022-03-08T22:52:21.231 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T22:52:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T22:52:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T22:52:21.238 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 516 Links: 1 2022-03-08T22:52:21.239 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T22:52:21.239 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2022-03-08 22:52:20.023143693 +0000 2022-03-08T22:52:21.239 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2022-03-08 22:52:20.023143693 +0000 2022-03-08T22:52:21.239 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2022-03-08 22:52:20.023143693 +0000 2022-03-08T22:52:21.240 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2022-03-08T22:52:21.246 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2022-03-08T22:52:21.247 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2022-03-08T22:52:21.247 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000519695 s, 985 kB/s 2022-03-08T22:52:21.251 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T22:52:21.335 INFO:teuthology.orchestra.run.smithi176.stdout:loop 2022-03-08T22:52:21.336 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_1... 2022-03-08T22:52:21.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T22:52:21.359 INFO:teuthology.orchestra.run.smithi176.stdout:1 2022-03-08T22:52:21.386 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_11 2022-03-08T22:52:21.410 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_2... 2022-03-08T22:52:21.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T22:52:21.472 INFO:teuthology.orchestra.run.smithi176.stdout:1 2022-03-08T22:52:21.500 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_21 2022-03-08T22:52:21.525 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_3... 2022-03-08T22:52:21.525 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T22:52:21.549 INFO:teuthology.orchestra.run.smithi176.stdout:1 2022-03-08T22:52:21.575 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_31 2022-03-08T22:52:21.599 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_4... 2022-03-08T22:52:21.599 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T22:52:21.618 INFO:teuthology.orchestra.run.smithi176.stdout:1 2022-03-08T22:52:21.643 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_41 2022-03-08T22:52:21.667 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:21.667 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:52:21.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme list 2022-03-08T22:52:21.730 INFO:teuthology.orchestra.run.smithi176.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T22:52:21.731 INFO:teuthology.orchestra.run.smithi176.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T22:52:21.731 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme0n1 CVFT62330099400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T22:52:21.731 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme1n1 344e42808f419485 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:21.731 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme2n1 6cfaa71eddeaa137 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:21.732 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme3n1 3581c8e3f405bebf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:21.732 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme4n1 085160b1a3d4f4d3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T22:52:21.733 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T22:52:21.733 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:21.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/scratch_devs 2022-03-08T22:52:21.749 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T22:52:21.820 INFO:tasks.rook:Rook start 2022-03-08T22:52:21.820 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4', 'spec': {'mon': {'allowMultiplePerNode': False}, 'network': {'provider': 'host'}}, 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-08T22:52:21.821 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:52:21.821 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-08T22:52:21.821 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-08T22:52:21.831 INFO:teuthology.orchestra.run.smithi176.stderr:Cloning into 'rook'... 2022-03-08T22:52:24.297 INFO:teuthology.orchestra.run.smithi176.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-08T22:52:24.298 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.298 INFO:teuthology.orchestra.run.smithi176.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-08T22:52:24.298 INFO:teuthology.orchestra.run.smithi176.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-08T22:52:24.298 INFO:teuthology.orchestra.run.smithi176.stderr:state without impacting any branches by switching back to a branch. 2022-03-08T22:52:24.299 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.299 INFO:teuthology.orchestra.run.smithi176.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-08T22:52:24.299 INFO:teuthology.orchestra.run.smithi176.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-08T22:52:24.299 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.300 INFO:teuthology.orchestra.run.smithi176.stderr: git switch -c 2022-03-08T22:52:24.300 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.300 INFO:teuthology.orchestra.run.smithi176.stderr:Or undo this operation with: 2022-03-08T22:52:24.300 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.300 INFO:teuthology.orchestra.run.smithi176.stderr: git switch - 2022-03-08T22:52:24.301 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.301 INFO:teuthology.orchestra.run.smithi176.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-08T22:52:24.301 INFO:teuthology.orchestra.run.smithi176.stderr: 2022-03-08T22:52:24.401 INFO:tasks.rook:/home/teuthworker 2022-03-08T22:52:24.401 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T22:52:24.407 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T22:52:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T22:52:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T22:52:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2022-03-08T22:52:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:24.408 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:52:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2022-03-08T22:52:24.409 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:52:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2022-03-08T22:52:24.410 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T22:52:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:24.410 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T22:52:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-08T22:52:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42897sec preferred_lft 42897sec 2022-03-08T22:52:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ec4:7aff:fe88:6f61/64 scope link 2022-03-08T22:52:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:24.412 INFO:teuthology.orchestra.run.smithi176.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T22:52:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 02:42:6f:7f:5b:c5 brd ff:ff:ff:ff:ff:ff 2022-03-08T22:52:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T22:52:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2022-03-08T22:52:24.413 DEBUG:teuthology.orchestra.run.smithi176:> which systemctl 2022-03-08T22:52:24.419 INFO:teuthology.orchestra.run.smithi176.stdout:/usr/bin/systemctl 2022-03-08T22:52:24.419 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T22:52:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T22:52:24.470 DEBUG:teuthology.orchestra.run.smithi176:> true 2022-03-08T22:52:24.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-03-08T22:52:24.559 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-03-08T22:52:24.564 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:52:24.565 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T22:52:24.578 DEBUG:teuthology.orchestra.remote:smithi176:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:52:24.591 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:24.591 DEBUG:teuthology.orchestra.run.smithi176:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-08T22:52:24.598 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-08T22:52:24.646 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:52:24.647 DEBUG:teuthology.orchestra.run.smithi176:> dd of=operator.yaml 2022-03-08T22:52:24.654 INFO:tasks.rook:Deploying operator 2022-03-08T22:52:24.675 DEBUG:teuthology.orchestra.remote:smithi176:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:52:24.740 DEBUG:teuthology.orchestra.remote:smithi176:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:52:24.753 DEBUG:teuthology.orchestra.run.smithi176:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-08T22:52:25.009 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T22:52:25.033 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T22:52:25.214 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T22:52:25.238 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T22:52:25.289 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T22:52:25.367 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T22:52:25.384 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T22:52:25.443 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T22:52:25.553 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T22:52:25.583 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T22:52:25.613 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T22:52:25.717 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T22:52:25.725 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T22:52:25.753 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T22:52:25.776 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-08T22:52:25.803 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-08T22:52:25.895 INFO:teuthology.orchestra.run.smithi176.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-08T22:52:25.978 INFO:teuthology.orchestra.run.smithi176.stdout:namespace/rook-ceph created 2022-03-08T22:52:26.162 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:52:26.164 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-08T22:52:26.182 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-08T22:52:26.659 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-08T22:52:26.661 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:52:26.927 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:26.940 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:26.959 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:52:26.982 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:52:27.076 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T22:52:27.079 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-ceph-system created 2022-03-08T22:52:27.100 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:27.324 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T22:52:27.328 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T22:52:27.357 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T22:52:27.391 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T22:52:27.527 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T22:52:27.568 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:27.611 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:27.702 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:52:27.705 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:52:27.760 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:52:27.776 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T22:52:27.781 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:27.797 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T22:52:27.850 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T22:52:27.853 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T22:52:27.865 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T22:52:27.908 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T22:52:27.934 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T22:52:27.934 INFO:teuthology.orchestra.run.smithi176.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T22:52:28.018 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T22:52:28.021 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T22:52:28.046 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T22:52:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T22:52:28.102 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T22:52:28.210 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T22:52:28.214 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T22:52:28.232 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T22:52:28.245 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T22:52:28.285 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T22:52:28.369 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:52:28.373 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T22:52:28.381 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T22:52:28.390 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T22:52:28.415 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T22:52:28.427 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T22:52:28.591 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T22:52:28.595 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T22:52:28.917 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T22:52:29.027 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T22:52:29.028 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:52:29.183 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T22:52:29.187 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T22:52:29.274 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T22:52:29.376 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T22:52:29.395 INFO:teuthology.orchestra.run.smithi176.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T22:52:29.425 INFO:teuthology.orchestra.run.smithi176.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:52:29.450 INFO:teuthology.orchestra.run.smithi176.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T22:52:29.465 INFO:teuthology.orchestra.run.smithi176.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T22:52:29.545 INFO:teuthology.orchestra.run.smithi176.stdout:configmap/rook-ceph-operator-config created 2022-03-08T22:52:29.551 INFO:teuthology.orchestra.run.smithi176.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T22:52:29.555 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:29.746 INFO:teuthology.orchestra.run.smithi176.stderr:No resources found in rook-ceph namespace. 2022-03-08T22:52:39.751 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:39.815 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:39.815 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 Pending 0 10s 2022-03-08T22:52:49.818 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:49.878 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:49.878 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 Pending 0 20s 2022-03-08T22:52:59.881 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:52:59.939 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:52:59.940 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 Pending 0 30s 2022-03-08T22:53:09.942 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:10.119 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:10.120 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 Pending 0 41s 2022-03-08T22:53:20.123 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:20.524 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:20.524 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 ContainerCreating 0 51s 2022-03-08T22:53:30.527 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:30.592 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:30.592 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 ContainerCreating 0 61s 2022-03-08T22:53:40.595 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 ContainerCreating 0 71s 2022-03-08T22:53:50.663 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:53:50.772 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:53:50.944 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 0/1 ContainerCreating 0 81s 2022-03-08T22:54:00.946 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T22:54:01.016 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:01.016 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-operator-6b496c8959-h8srf 1/1 Running 0 92s 2022-03-08T22:54:01.018 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-h8srf 2022-03-08T22:54:01.027 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:54:01.028 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T22:54:01.080 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-08T22:54:01.081 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T22:54:01.081 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T22:54:01.081 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T22:54:01.082 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T22:54:01.082 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T22:54:01.082 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T22:54:01.083 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T22:54:01.083 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T22:54:01.086 DEBUG:teuthology.orchestra.run.smithi176:> kubectl create -f - 2022-03-08T22:54:01.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.081478 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T22:54:01.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.081577 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-08T22:54:01.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.081581 I | cephcmd: starting Rook-Ceph operator 2022-03-08T22:54:01.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.223420 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-08T22:54:01.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.228926 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T22:54:01.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.230768 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T22:54:01.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.233181 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-08T22:54:01.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.234690 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T22:54:01.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.237194 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T22:54:01.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.237208 I | operator: watching all namespaces for ceph cluster CRs 2022-03-08T22:54:01.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.237344 I | operator: setting up the controller-runtime manager 2022-03-08T22:54:01.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.239191 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-08T22:54:01.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.244736 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T22:54:01.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644548 I | ceph-cluster-controller: successfully started 2022-03-08T22:54:01.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644642 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T22:54:01.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644667 I | ceph-crashcollector-controller: successfully started 2022-03-08T22:54:01.112 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644746 I | ceph-block-pool-controller: successfully started 2022-03-08T22:54:01.112 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644818 I | ceph-object-store-user-controller: successfully started 2022-03-08T22:54:01.112 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644895 I | ceph-object-realm-controller: successfully started 2022-03-08T22:54:01.112 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.644962 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T22:54:01.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645021 I | ceph-object-zone-controller: successfully started 2022-03-08T22:54:01.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645164 I | ceph-object-controller: successfully started 2022-03-08T22:54:01.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645274 I | ceph-file-controller: successfully started 2022-03-08T22:54:01.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645363 I | ceph-nfs-controller: successfully started 2022-03-08T22:54:01.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645446 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T22:54:01.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645527 I | ceph-client-controller: successfully started 2022-03-08T22:54:01.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.645597 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T22:54:01.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:53:54.646836 I | operator: starting the controller-runtime manager 2022-03-08T22:54:02.076 INFO:teuthology.orchestra.run.smithi176.stdout:configmap/rook-config-override created 2022-03-08T22:54:02.082 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: false count: 1 network: provider: host skipUpgradeChecks: true 2022-03-08T22:54:02.082 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:54:02.082 DEBUG:teuthology.orchestra.run.smithi176:> dd of=cluster.yaml 2022-03-08T22:54:02.128 DEBUG:teuthology.orchestra.run.smithi176:> kubectl create -f cluster.yaml 2022-03-08T22:54:04.482 INFO:teuthology.orchestra.run.smithi176.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T22:54:04.487 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.488597 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-08T22:54:04.487 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.488767 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T22:54:04.498 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.499690 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:54:04.504 DEBUG:teuthology.orchestra.remote:smithi176:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T22:54:04.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.507633 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-08T22:54:04.516 DEBUG:teuthology.orchestra.run.smithi176:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T22:54:04.520 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.522089 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T22:54:04.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.586043 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-08T22:54:04.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.586101 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T22:54:04.588 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.588891 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T22:54:04.591 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.591076 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T22:54:04.592 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.593294 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T22:54:04.595 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.596433 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T22:54:04.598 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.598872 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-08T22:54:04.712 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.713354 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:54:04.761 INFO:teuthology.orchestra.run.smithi176.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T22:54:04.766 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T22:54:04.767 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:04.865 INFO:teuthology.orchestra.run.smithi176.stderr:No resources found in rook-ceph namespace. 2022-03-08T22:54:04.917 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:04.917217 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-08T22:54:05.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:05.319291 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-08T22:54:05.841 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:05.842936 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-08T22:54:05.934 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:05.935829 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-08T22:54:06.139 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:06.139433 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T22:54:06.302 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:06.303917 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-08T22:54:06.526 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:06.527587 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T22:54:06.703 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:06.704908 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T22:54:06.704 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:06.704943 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-08T22:54:06.924 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:06.925915 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:09.869 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:10.043 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:10.043 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 6s 2022-03-08T22:54:15.046 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:15.110 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:15.110 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 11s 2022-03-08T22:54:20.112 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:20.176 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:20.176 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 16s 2022-03-08T22:54:25.178 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:25.238 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:25.238 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 21s 2022-03-08T22:54:30.241 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:30.441 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:30.442 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 26s 2022-03-08T22:54:35.445 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:35.914 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:35.915 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 31s 2022-03-08T22:54:40.918 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:40.975 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:40.976 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 36s 2022-03-08T22:54:42.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.568498 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-08T22:54:42.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.662116 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T22:54:42.709 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.709377 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T22:54:42.714 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.709564 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:54:42.714 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.710646 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:54:42.715 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.710691 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:54:42.760 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.758757 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:54:42.760 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.758914 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T22:54:42.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.843802 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T22:54:42.876 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.876026 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T22:54:42.879 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.877501 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 22:54:42.850832947 +0000 UTC m=+48.840198025 LastTransitionTime:2022-03-08 22:54:42.850832847 +0000 UTC m=+48.840197924}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:54:42.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.909073 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:54:42.916 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.916955 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T22:54:42.920 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.919167 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T22:54:42.925 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:42.925154 I | op-mon: start running mons 2022-03-08T22:54:43.026 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:43.026046 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T22:54:43.275 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:43.275553 I | op-mon: creating mon secrets for a new cluster 2022-03-08T22:54:43.411 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:43.411209 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T22:54:43.812 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:43.812511 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T22:54:44.255 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:44.253167 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:54:44.413 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:44.413102 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T22:54:44.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:44.619557 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T22:54:44.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:44.619587 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T22:54:44.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:44.814176 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T22:54:45.014 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:45.014029 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-08T22:54:45.413 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:45.412772 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T22:54:45.978 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:46.038 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:46.039 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 42s 2022-03-08T22:54:46.211 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:46.211571 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:46.412 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:46.412145 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:54:46.413 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:46.412380 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:54:46.634 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:46.634593 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:47.012 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:47.012575 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:54:47.412 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:47.412562 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:54:47.812 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:47.812696 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:54:48.214 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:48.214316 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:54:48.627 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:48.626699 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T22:54:48.827 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:48.827423 I | op-mon: targeting the mon count 1 2022-03-08T22:54:48.904 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:48.903975 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T22:54:49.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.052911 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:49.517 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.516134 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:49.841 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.841701 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:54:49.846 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.844258 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi176 2022-03-08T22:54:49.846 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.844278 I | op-mon: assignmon: mon a assigned to node smithi176 2022-03-08T22:54:49.850 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.848196 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T22:54:49.856 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.856841 I | op-mon: creating mon a 2022-03-08T22:54:49.857 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:49.856877 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-08T22:54:50.130 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:50.128321 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T22:54:50.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:50.928755 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T22:54:50.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:50.930007 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T22:54:50.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:50.931036 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:-1] 2022-03-08T22:54:50.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:50.932214 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T22:54:51.041 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:51.135 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:51.135 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 47s 2022-03-08T22:54:51.226 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:51.226864 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T22:54:51.627 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:51.626925 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T22:54:51.974 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:51.974721 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:54:51.978 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:51.975000 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:54:52.258 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:52.257086 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T22:54:52.493 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:52.492943 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T22:54:52.869 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:52.869337 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T22:54:53.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:53.099146 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T22:54:53.104 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:53.100587 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T22:54:53.579 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:53.578675 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T22:54:53.804 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:53.802591 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T22:54:54.069 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.069666 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T22:54:54.332 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.330036 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T22:54:54.333 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.330060 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:54:54.861 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.861322 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-08T22:54:54.863 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.861907 I | op-mon: mon a is not yet running 2022-03-08T22:54:54.864 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.861924 I | op-mon: mons running: [] 2022-03-08T22:54:54.866 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:54:54.866683 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T22:54:56.138 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:54:56.189 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:54:56.190 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 52s 2022-03-08T22:55:01.192 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:01.643 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:01.644 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 57s 2022-03-08T22:55:06.646 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:06.950 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:06.950 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 62s 2022-03-08T22:55:11.952 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:12.152 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:12.152 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 0/1 ContainerCreating 0 68s 2022-03-08T22:55:14.974 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:14.974558 I | op-mon: mon a is not yet running 2022-03-08T22:55:14.974 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:14.974583 I | op-mon: mons running: [] 2022-03-08T22:55:17.154 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T22:55:17.215 INFO:teuthology.orchestra.run.smithi176.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T22:55:17.216 INFO:teuthology.orchestra.run.smithi176.stdout:rook-ceph-tools-7677f7f4b9-4qgtx 1/1 Running 0 73s 2022-03-08T22:55:17.218 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T22:55:17.219 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status -f json 2022-03-08T22:55:18.080 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:55:18.081 INFO:teuthology.orchestra.run.smithi176.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:18.081 INFO:teuthology.orchestra.run.smithi176.stderr:pg stat 2022-03-08T22:55:18.081 INFO:teuthology.orchestra.run.smithi176.stderr:pg getmap 2022-03-08T22:55:18.081 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:18.082 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:18.082 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_pools_json 2022-03-08T22:55:18.082 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-pool [...] 2022-03-08T22:55:18.082 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:18.083 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:18.083 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls [] [...] 2022-03-08T22:55:18.083 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:18.083 INFO:teuthology.orchestra.run.smithi176.stderr:Error EINVAL: invalid command 2022-03-08T22:55:18.084 INFO:teuthology.orchestra.run.smithi176.stderr:command terminated with exit code 22 2022-03-08T22:55:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:18.187 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:18.187143 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:18.187 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:18.187173 I | op-mon: mons created: 1 2022-03-08T22:55:18.398 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:18.398886 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:55:18.403 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:18.404257 I | op-mon: mons running: [a] 2022-03-08T22:55:18.632 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:18.632943 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:55:18.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:18.632984 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:19.200 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:19.199876 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:55:19.200 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:19.199900 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:19.747 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:19.746790 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:55:19.747 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:19.746827 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:20.106 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:20.106611 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:55:20.107 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:20.106634 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:20.808 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:20.808685 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:55:20.809 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:20.808714 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:55:21.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:21.471760 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:55:21.472 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:21.471785 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:55:21.683 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:21.683116 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:55:21.683 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:21.683152 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:55:22.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:22.301283 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:55:22.903 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:22.902961 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:55:23.552 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:23.552698 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:55:24.304 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:24.304544 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:55:24.304 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:24.304574 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:55:24.606 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:24.606125 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:55:24.894 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:24.894631 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:55:24.895 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:24.894656 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:55:24.895 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:24.894660 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:55:25.178 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:25.178651 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:55:25.193 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:25.193173 I | op-mgr: start running mgr 2022-03-08T22:55:25.193 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:25.193203 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:55:25.721 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:25.721353 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi176": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:55:26.490 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:26.490503 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi176": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T22:55:28.089 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status -f json 2022-03-08T22:55:28.700 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:55:28.701 INFO:teuthology.orchestra.run.smithi176.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:28.701 INFO:teuthology.orchestra.run.smithi176.stderr:pg stat 2022-03-08T22:55:28.701 INFO:teuthology.orchestra.run.smithi176.stderr:pg getmap 2022-03-08T22:55:28.701 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:28.702 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:28.702 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_pools_json 2022-03-08T22:55:28.702 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-pool [...] 2022-03-08T22:55:28.702 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:28.703 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:28.703 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls [] [...] 2022-03-08T22:55:28.703 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:28.703 INFO:teuthology.orchestra.run.smithi176.stderr:Error EINVAL: invalid command 2022-03-08T22:55:28.704 INFO:teuthology.orchestra.run.smithi176.stderr:command terminated with exit code 22 2022-03-08T22:55:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:31.705 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.705871 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T22:55:31.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.708965 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:55:31.939 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.939036 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:55:31.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.939273 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T22:55:31.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.939362 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:55:31.946 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.947213 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:55:31.952 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.952620 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T22:55:31.955 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.953815 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:55:31.955 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:31.953888 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:55:32.426 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.356596 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T22:55:32.426 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.356612 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:55:32.426 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.362150 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:55:32.426 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.362178 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:55:32.427 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.362195 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T22:55:32.510 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.510567 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-08T22:55:32.525 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.525380 I | op-mgr: successful modules: prometheus 2022-03-08T22:55:32.600 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.597682 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T22:55:32.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.597700 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:55:32.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:32.598539 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-08T22:55:32.601 INFO:tasks.rook.operator.smithi176.stdout:I0308 22:55:32.598633 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T22:55:33.080 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:33.080907 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:33.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:33.769234 I | op-mgr: successful modules: dashboard 2022-03-08T22:55:37.751 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:37.751593 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-08T22:55:38.374 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:38.374946 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:38.707 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status -f json 2022-03-08T22:55:39.251 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:55:39.251 INFO:teuthology.orchestra.run.smithi176.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:39.252 INFO:teuthology.orchestra.run.smithi176.stderr:pg stat 2022-03-08T22:55:39.252 INFO:teuthology.orchestra.run.smithi176.stderr:pg getmap 2022-03-08T22:55:39.252 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:39.253 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:39.253 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_pools_json 2022-03-08T22:55:39.253 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-pool [...] 2022-03-08T22:55:39.253 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:39.253 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:39.254 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls [] [...] 2022-03-08T22:55:39.254 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:39.254 INFO:teuthology.orchestra.run.smithi176.stderr:Error EINVAL: invalid command 2022-03-08T22:55:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:39.256 INFO:teuthology.orchestra.run.smithi176.stderr:command terminated with exit code 22 2022-03-08T22:55:43.112 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:43.112601 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:55:43.670 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:43.670354 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:44.625 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:44.626304 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T22:55:48.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:48.961695 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-08T22:55:49.258 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status -f json 2022-03-08T22:55:49.727 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:55:49.727 INFO:teuthology.orchestra.run.smithi176.stderr:no valid command found; 10 closest matches: 2022-03-08T22:55:49.728 INFO:teuthology.orchestra.run.smithi176.stderr:pg stat 2022-03-08T22:55:49.728 INFO:teuthology.orchestra.run.smithi176.stderr:pg getmap 2022-03-08T22:55:49.728 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T22:55:49.728 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T22:55:49.729 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_pools_json 2022-03-08T22:55:49.729 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-pool [...] 2022-03-08T22:55:49.729 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-primary [] [...] 2022-03-08T22:55:49.729 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls-by-osd [] [...] 2022-03-08T22:55:49.729 INFO:teuthology.orchestra.run.smithi176.stderr:pg ls [] [...] 2022-03-08T22:55:49.730 INFO:teuthology.orchestra.run.smithi176.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T22:55:49.730 INFO:teuthology.orchestra.run.smithi176.stderr:Error EINVAL: invalid command 2022-03-08T22:55:49.732 INFO:teuthology.orchestra.run.smithi176.stderr:command terminated with exit code 22 2022-03-08T22:55:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T22:55:49.914 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:49.915001 I | cephclient: command failed for set rook backend. trying again... 2022-03-08T22:55:54.383 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:54.383737 I | op-mgr: successful modules: balancer 2022-03-08T22:55:55.333 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:55.334306 I | cephclient: action set rook backend succeeded on attempt 2 2022-03-08T22:55:55.334 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:55:55.334330 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:55:59.735 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status -f json 2022-03-08T22:56:00.312 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:00.313 INFO:teuthology.orchestra.run.smithi176.stdout:{"available": true, "backend": "rook"} 2022-03-08T22:56:00.336 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T22:56:00.336 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T22:56:00.940 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch apply osd --all-available-devices 2022-03-08T22:56:01.571 INFO:teuthology.orchestra.run.smithi176.stdout:['Success'] 2022-03-08T22:56:01.574 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:01.573685 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T22:56:01.574 INFO:tasks.rook.operator.smithi176.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T22:56:01.574 INFO:tasks.rook.operator.smithi176.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T22:56:01.574 INFO:tasks.rook.operator.smithi176.stdout:   ... // 3 identical fields 2022-03-08T22:56:01.575 INFO:tasks.rook.operator.smithi176.stdout:   Config: nil, 2022-03-08T22:56:01.575 INFO:tasks.rook.operator.smithi176.stdout:   Selection: {}, 2022-03-08T22:56:01.575 INFO:tasks.rook.operator.smithi176.stdout:-  StorageClassDeviceSets: nil, 2022-03-08T22:56:01.575 INFO:tasks.rook.operator.smithi176.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T22:56:01.576 INFO:tasks.rook.operator.smithi176.stdout:+  { 2022-03-08T22:56:01.576 INFO:tasks.rook.operator.smithi176.stdout:+  Name: "smithi176-lv-1", 2022-03-08T22:56:01.576 INFO:tasks.rook.operator.smithi176.stdout:+  Count: 1, 2022-03-08T22:56:01.576 INFO:tasks.rook.operator.smithi176.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.576 INFO:tasks.rook.operator.smithi176.stdout:+  }, 2022-03-08T22:56:01.577 INFO:tasks.rook.operator.smithi176.stdout:+  { 2022-03-08T22:56:01.577 INFO:tasks.rook.operator.smithi176.stdout:+  Name: "smithi176-lv-2", 2022-03-08T22:56:01.578 INFO:tasks.rook.operator.smithi176.stdout:+  Count: 1, 2022-03-08T22:56:01.578 INFO:tasks.rook.operator.smithi176.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.579 INFO:tasks.rook.operator.smithi176.stdout:+  }, 2022-03-08T22:56:01.579 INFO:tasks.rook.operator.smithi176.stdout:+  { 2022-03-08T22:56:01.579 INFO:tasks.rook.operator.smithi176.stdout:+  Name: "smithi176-lv-3", 2022-03-08T22:56:01.579 INFO:tasks.rook.operator.smithi176.stdout:+  Count: 1, 2022-03-08T22:56:01.580 INFO:tasks.rook.operator.smithi176.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.580 INFO:tasks.rook.operator.smithi176.stdout:+  }, 2022-03-08T22:56:01.580 INFO:tasks.rook.operator.smithi176.stdout:+  { 2022-03-08T22:56:01.580 INFO:tasks.rook.operator.smithi176.stdout:+  Name: "smithi176-lv-4", 2022-03-08T22:56:01.580 INFO:tasks.rook.operator.smithi176.stdout:+  Count: 1, 2022-03-08T22:56:01.581 INFO:tasks.rook.operator.smithi176.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T22:56:01.581 INFO:tasks.rook.operator.smithi176.stdout:+  }, 2022-03-08T22:56:01.581 INFO:tasks.rook.operator.smithi176.stdout:+  }, 2022-03-08T22:56:01.581 INFO:tasks.rook.operator.smithi176.stdout:   }, 2022-03-08T22:56:01.581 INFO:tasks.rook.operator.smithi176.stdout:   Annotations: nil, 2022-03-08T22:56:01.582 INFO:tasks.rook.operator.smithi176.stdout:   Labels: nil, 2022-03-08T22:56:01.582 INFO:tasks.rook.operator.smithi176.stdout:   ... // 20 identical fields 2022-03-08T22:56:01.582 INFO:tasks.rook.operator.smithi176.stdout:  } 2022-03-08T22:56:01.582 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:01.573765 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:56:01.594 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-08T22:56:01.594 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:56:01.602 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:01.602971 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T22:56:01.634 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:01.635414 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:56:02.163 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:02.180 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":4,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:02.181 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:07.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.099387 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:56:07.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.099419 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:56:07.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.099433 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:56:07.397 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.397438 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T22:56:07.433 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.433539 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:07.433 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.433742 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:07.868 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:07.868657 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:56:08.022 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:08.022300 I | op-mon: start running mons 2022-03-08T22:56:08.047 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:08.047862 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T22:56:08.074 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:08.074710 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T22:56:08.080 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:08.081135 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:08.080 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:08.081250 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:09.023 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:09.024208 I | op-mon: targeting the mon count 1 2022-03-08T22:56:09.066 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:09.066799 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T22:56:10.067 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:10.067889 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:56:10.074 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:10.075931 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T22:56:10.091 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:10.092764 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T22:56:10.141 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:10.142354 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T22:56:10.148 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:10.149179 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T22:56:12.155 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:12.155357 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T22:56:12.182 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:56:12.646 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:12.665 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":4,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:12.665 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:17.740 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:17.738552 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T22:56:17.740 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:17.738571 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T22:56:17.741 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:17.738584 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T22:56:17.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:17.853096 I | op-mon: checking if multiple mons are on the same node 2022-03-08T22:56:17.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:17.853227 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T22:56:18.262 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:18.263131 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:18.265 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:18.263462 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:18.744 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:18.744661 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T22:56:18.872 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:18.872861 I | op-mon: start running mons 2022-03-08T22:56:18.914 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:18.914727 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T22:56:19.261 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:19.262014 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T22:56:19.521 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:19.294213 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:19.521 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:19.294392 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:19.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:19.930264 I | op-mon: targeting the mon count 1 2022-03-08T22:56:19.933 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:19.934117 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:56:20.262 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:20.263027 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T22:56:20.262 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:20.263061 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:56:20.607 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:20.607648 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T22:56:20.607 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:20.607680 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:56:20.973 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:20.973986 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T22:56:20.973 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:20.974018 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:56:21.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:21.331000 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T22:56:21.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:21.331034 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:56:21.686 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:21.687687 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T22:56:21.687 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:21.687707 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T22:56:22.015 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.015704 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T22:56:22.015 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.015730 I | op-mon: checking for basic quorum with existing mons 2022-03-08T22:56:22.015 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.015742 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-08T22:56:22.056 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.057022 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T22:56:22.062 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.063416 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T22:56:22.062 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.063653 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T22:56:22.078 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.079402 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T22:56:22.085 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.086583 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T22:56:22.085 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.086606 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:56:22.089 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.090859 I | op-mon: mons running: [a] 2022-03-08T22:56:22.489 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.489756 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:56:22.489 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.489788 I | op-mon: mons created: 1 2022-03-08T22:56:22.666 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:56:22.914 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.915659 I | op-mon: waiting for mon quorum with [a] 2022-03-08T22:56:22.921 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:22.922444 I | op-mon: mons running: [a] 2022-03-08T22:56:23.176 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:23.200 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":4,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:23.201 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:23.354 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:23.355205 I | op-mon: Monitors in quorum: [a] 2022-03-08T22:56:23.355 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:23.355248 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T22:56:23.778 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:23.778895 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T22:56:24.194 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:24.195516 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T22:56:24.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:24.633688 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T22:56:25.063 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:25.063904 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T22:56:25.063 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:25.063928 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T22:56:25.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:25.482724 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T22:56:25.904 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:25.905479 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T22:56:25.905 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:25.905513 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T22:56:25.905 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:25.905520 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T22:56:26.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.318045 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T22:56:26.345 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.346196 I | op-mgr: start running mgr 2022-03-08T22:56:26.345 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.346241 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T22:56:26.813 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.814163 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T22:56:26.821 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.822075 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T22:56:26.823 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.824638 I | op-mgr: setting services to point to mgr "a" 2022-03-08T22:56:26.940 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.940790 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T22:56:26.940 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.940815 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T22:56:26.956 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:26.957064 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T22:56:27.051 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.051720 I | op-osd: creating 1 new PVCs for device set "smithi176-lv-1" 2022-03-08T22:56:27.260 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.256303 I | op-osd: successfully provisioned PVC "smithi176-lv-1-data-0qcsq9" 2022-03-08T22:56:27.260 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.256336 I | op-osd: creating 1 new PVCs for device set "smithi176-lv-2" 2022-03-08T22:56:27.452 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.452815 I | op-osd: successfully provisioned PVC "smithi176-lv-2-data-0spfs6" 2022-03-08T22:56:27.452 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.452850 I | op-osd: creating 1 new PVCs for device set "smithi176-lv-3" 2022-03-08T22:56:27.654 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.652918 I | op-osd: successfully provisioned PVC "smithi176-lv-3-data-0w78wp" 2022-03-08T22:56:27.654 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.652959 I | op-osd: creating 1 new PVCs for device set "smithi176-lv-4" 2022-03-08T22:56:27.854 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.855329 I | op-osd: successfully provisioned PVC "smithi176-lv-4-data-0dsmg4" 2022-03-08T22:56:27.886 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.887430 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-1-data-0qcsq9" 2022-03-08T22:56:27.930 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:27.931994 I | op-mgr: successful modules: balancer 2022-03-08T22:56:28.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.268899 I | op-osd: started OSD provisioning job for PVC "smithi176-lv-1-data-0qcsq9" 2022-03-08T22:56:28.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.268927 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-2-data-0spfs6" 2022-03-08T22:56:28.422 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.422880 I | op-mgr: successful modules: prometheus 2022-03-08T22:56:28.424 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.423900 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T22:56:28.424 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.424293 I | op-mgr: successful modules: dashboard 2022-03-08T22:56:28.674 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.672836 I | op-osd: started OSD provisioning job for PVC "smithi176-lv-2-data-0spfs6" 2022-03-08T22:56:28.675 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:28.673119 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-3-data-0w78wp" 2022-03-08T22:56:29.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:29.329114 I | op-osd: started OSD provisioning job for PVC "smithi176-lv-3-data-0w78wp" 2022-03-08T22:56:29.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:29.329152 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-4-data-0dsmg4" 2022-03-08T22:56:29.756 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:29.756549 I | op-osd: started OSD provisioning job for PVC "smithi176-lv-4-data-0dsmg4" 2022-03-08T22:56:29.756 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:29.756580 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T22:56:29.756 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:29.756587 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T22:56:30.149 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:30.150221 I | op-osd: OSD orchestration status for PVC smithi176-lv-1-data-0qcsq9 is "starting" 2022-03-08T22:56:30.149 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:30.150506 I | op-osd: OSD orchestration status for PVC smithi176-lv-2-data-0spfs6 is "starting" 2022-03-08T22:56:30.153 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:30.150727 I | op-osd: OSD orchestration status for PVC smithi176-lv-3-data-0w78wp is "starting" 2022-03-08T22:56:30.154 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:30.150938 I | op-osd: OSD orchestration status for PVC smithi176-lv-4-data-0dsmg4 is "starting" 2022-03-08T22:56:30.581 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:30.582539 I | op-mgr: successful modules: orchestrator modules 2022-03-08T22:56:33.202 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:56:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:33.684 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T22:56:33.684 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:39.082 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:39.083059 I | op-osd: OSD orchestration status for node smithi176-lv-1-data-0qcsq9 is "orchestrating" 2022-03-08T22:56:39.737 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:39.737703 I | op-osd: OSD orchestration status for PVC smithi176-lv-1-data-0qcsq9 is "orchestrating" 2022-03-08T22:56:42.612 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:42.612881 I | op-osd: OSD orchestration status for node smithi176-lv-3-data-0w78wp is "orchestrating" 2022-03-08T22:56:42.619 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:42.620944 I | op-osd: OSD orchestration status for node smithi176-lv-4-data-0dsmg4 is "orchestrating" 2022-03-08T22:56:42.959 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:42.960780 I | op-osd: OSD orchestration status for PVC smithi176-lv-3-data-0w78wp is "orchestrating" 2022-03-08T22:56:43.148 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:43.146074 I | op-osd: OSD orchestration status for PVC smithi176-lv-4-data-0dsmg4 is "orchestrating" 2022-03-08T22:56:43.686 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:56:44.174 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:44.192 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":6,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1646780201,"num_remapped_pgs":0} 2022-03-08T22:56:44.193 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:46.785 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:46.786442 I | op-osd: OSD orchestration status for PVC smithi176-lv-1-data-0qcsq9 is "completed" 2022-03-08T22:56:46.786 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:46.786469 I | op-osd: creating OSD 0 on PVC "smithi176-lv-1-data-0qcsq9" 2022-03-08T22:56:46.786 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:46.786480 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-1-data-0qcsq9" 2022-03-08T22:56:49.721 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:49.722076 I | op-osd: OSD orchestration status for PVC smithi176-lv-3-data-0w78wp is "completed" 2022-03-08T22:56:49.721 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:49.722119 I | op-osd: creating OSD 1 on PVC "smithi176-lv-3-data-0w78wp" 2022-03-08T22:56:49.722 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:49.722138 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-3-data-0w78wp" 2022-03-08T22:56:50.649 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:50.645919 I | op-osd: OSD orchestration status for PVC smithi176-lv-4-data-0dsmg4 is "completed" 2022-03-08T22:56:50.649 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:50.645950 I | op-osd: creating OSD 2 on PVC "smithi176-lv-4-data-0dsmg4" 2022-03-08T22:56:50.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:50.645960 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-4-data-0dsmg4" 2022-03-08T22:56:54.194 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:56:55.012 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:56:55.030 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":8,"num_osds":3,"num_up_osds":0,"osd_up_since":0,"num_in_osds":3,"osd_in_since":1646780204,"num_remapped_pgs":0} 2022-03-08T22:56:55.031 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:56:57.939 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:57.940421 I | op-osd: OSD orchestration status for node smithi176-lv-2-data-0spfs6 is "orchestrating" 2022-03-08T22:56:58.811 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:56:58.813092 I | op-osd: OSD orchestration status for PVC smithi176-lv-2-data-0spfs6 is "orchestrating" 2022-03-08T22:57:05.032 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:57:05.690 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:57:05.691724 I | op-osd: OSD orchestration status for PVC smithi176-lv-2-data-0spfs6 is "completed" 2022-03-08T22:57:05.693 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:57:05.692253 I | op-osd: creating OSD 3 on PVC "smithi176-lv-2-data-0spfs6" 2022-03-08T22:57:05.693 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:57:05.692274 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-2-data-0spfs6" 2022-03-08T22:57:06.747 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:06.767 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":9,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780220,"num_remapped_pgs":0} 2022-03-08T22:57:06.767 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:57:07.845 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:57:07.846469 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T22:57:07.845 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 22:57:07.846487 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T22:57:16.768 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:57:17.759 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:17.783 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":9,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646780220,"num_remapped_pgs":0} 2022-03-08T22:57:17.783 INFO:tasks.rook: have 0/4 OSDs 2022-03-08T22:57:27.785 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:57:28.258 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:28.278 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":14,"num_osds":4,"num_up_osds":3,"osd_up_since":1646780246,"num_in_osds":4,"osd_in_since":1646780220,"num_remapped_pgs":0} 2022-03-08T22:57:28.278 INFO:tasks.rook: have 3/4 OSDs 2022-03-08T22:57:38.279 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd stat -f json 2022-03-08T22:57:38.794 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:38.812 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":20,"num_osds":4,"num_up_osds":4,"osd_up_since":1646780253,"num_in_osds":4,"osd_in_since":1646780220,"num_remapped_pgs":0} 2022-03-08T22:57:38.812 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T22:57:38.813 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- cat /etc/ceph/ceph.conf 2022-03-08T22:57:38.975 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2022-03-08T22:57:38.975 INFO:teuthology.orchestra.run.smithi176.stdout:mon_host = 172.21.15.176:6789 2022-03-08T22:57:38.975 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:38.976 INFO:teuthology.orchestra.run.smithi176.stdout:[client.admin] 2022-03-08T22:57:38.976 INFO:teuthology.orchestra.run.smithi176.stdout:keyring = /etc/ceph/keyring 2022-03-08T22:57:38.978 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- cat /etc/ceph/keyring 2022-03-08T22:57:39.203 INFO:teuthology.orchestra.run.smithi176.stdout:[client.admin] 2022-03-08T22:57:39.204 INFO:teuthology.orchestra.run.smithi176.stdout:key = AQCz3idi7MxWEBAAcSYzd0aEaqr3UQFpVbDvww== 2022-03-08T22:57:39.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2022-03-08T22:57:39.219 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:57:39.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T22:57:39.232 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:57:39.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/keyring 2022-03-08T22:57:39.243 INFO:tasks.rook:Setting up client nodes... 2022-03-08T22:57:39.243 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T22:57:39.883 INFO:teuthology.orchestra.run.smithi176.stdout:[client.a] 2022-03-08T22:57:39.884 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBj3ydiuOTAMRAA7gH6QUafv3UOAGdeM2fbvQ== 2022-03-08T22:57:39.904 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:57:39.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T22:57:39.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T22:57:39.926 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd dump --format=json 2022-03-08T22:57:40.415 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:40.416 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":20,"fsid":"b6225762-85ac-4d55-aa25-1c146868c4f1","created":"2022-03-08T22:55:16.768945+0000","modified":"2022-03-08T22:57:34.749672+0000","last_up_change":"2022-03-08T22:57:33.640261+0000","last_in_change":"2022-03-08T22:57:00.615232+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T22:57:29.218823+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8f69b98f-f349-4192-b4f4-4d8f51a04951","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6811","nonce":3750486074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6813","nonce":3750486074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6817","nonce":3750486074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6815","nonce":3750486074}]},"public_addr":"172.21.15.176:6811/3750486074","cluster_addr":"172.21.15.176:6813/3750486074","heartbeat_back_addr":"172.21.15.176:6817/3750486074","heartbeat_front_addr":"172.21.15.176:6815/3750486074","state":["exists","up"]},{"osd":1,"uuid":"3c1dc57b-3e37-4862-9d1e-8185d624d2c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6803","nonce":2786447556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6805","nonce":2786447556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6809","nonce":2786447556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6807","nonce":2786447556}]},"public_addr":"172.21.15.176:6803/2786447556","cluster_addr":"172.21.15.176:6805/2786447556","heartbeat_back_addr":"172.21.15.176:6809/2786447556","heartbeat_front_addr":"172.21.15.176:6807/2786447556","state":["exists","up"]},{"osd":2,"uuid":"20e74f9b-ac2e-4200-8196-38a3b780c9b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6819","nonce":3585544879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6821","nonce":3585544879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6825","nonce":3585544879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6823","nonce":3585544879}]},"public_addr":"172.21.15.176:6819/3585544879","cluster_addr":"172.21.15.176:6821/3585544879","heartbeat_back_addr":"172.21.15.176:6825/3585544879","heartbeat_front_addr":"172.21.15.176:6823/3585544879","state":["exists","up"]},{"osd":3,"uuid":"2db5663d-d81d-4a8e-b31d-b1812c53d68a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6827","nonce":1763722339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6829","nonce":1763722339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6833","nonce":1763722339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6831","nonce":1763722339}]},"public_addr":"172.21.15.176:6827/1763722339","cluster_addr":"172.21.15.176:6829/1763722339","heartbeat_back_addr":"172.21.15.176:6833/1763722339","heartbeat_front_addr":"172.21.15.176:6831/1763722339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:23.124937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:22.444308+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:25.014164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:32.163410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T22:57:40.437 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T22:57:29.218823+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T22:57:40.438 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd pool get .mgr pg_num 2022-03-08T22:57:41.003 INFO:teuthology.orchestra.run.smithi176.stdout:pg_num: 1 2022-03-08T22:57:41.023 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T22:57:41.024 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T22:57:41.024 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph mgr dump --format=json 2022-03-08T22:57:41.556 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:41.575 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":13,"active_gid":4170,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3162345799},{"type":"v1","addr":"172.21.15.176:6801","nonce":3162345799}]},"active_addr":"172.21.15.176:6801/3162345799","active_change":"2022-03-08T22:55:51.973901+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://172.21.15.176:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":1289490480}]},{"addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":2493782587}]},{"addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":2900805743}]},{"addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":1600333155}]}]}} 2022-03-08T22:57:41.578 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T22:57:41.579 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T22:57:41.579 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd dump --format=json 2022-03-08T22:57:42.105 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:42.105 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":20,"fsid":"b6225762-85ac-4d55-aa25-1c146868c4f1","created":"2022-03-08T22:55:16.768945+0000","modified":"2022-03-08T22:57:34.749672+0000","last_up_change":"2022-03-08T22:57:33.640261+0000","last_in_change":"2022-03-08T22:57:00.615232+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T22:57:29.218823+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8f69b98f-f349-4192-b4f4-4d8f51a04951","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6811","nonce":3750486074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6813","nonce":3750486074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6817","nonce":3750486074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6815","nonce":3750486074}]},"public_addr":"172.21.15.176:6811/3750486074","cluster_addr":"172.21.15.176:6813/3750486074","heartbeat_back_addr":"172.21.15.176:6817/3750486074","heartbeat_front_addr":"172.21.15.176:6815/3750486074","state":["exists","up"]},{"osd":1,"uuid":"3c1dc57b-3e37-4862-9d1e-8185d624d2c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6803","nonce":2786447556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6805","nonce":2786447556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6809","nonce":2786447556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6807","nonce":2786447556}]},"public_addr":"172.21.15.176:6803/2786447556","cluster_addr":"172.21.15.176:6805/2786447556","heartbeat_back_addr":"172.21.15.176:6809/2786447556","heartbeat_front_addr":"172.21.15.176:6807/2786447556","state":["exists","up"]},{"osd":2,"uuid":"20e74f9b-ac2e-4200-8196-38a3b780c9b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6819","nonce":3585544879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6821","nonce":3585544879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6825","nonce":3585544879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6823","nonce":3585544879}]},"public_addr":"172.21.15.176:6819/3585544879","cluster_addr":"172.21.15.176:6821/3585544879","heartbeat_back_addr":"172.21.15.176:6825/3585544879","heartbeat_front_addr":"172.21.15.176:6823/3585544879","state":["exists","up"]},{"osd":3,"uuid":"2db5663d-d81d-4a8e-b31d-b1812c53d68a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6827","nonce":1763722339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6829","nonce":1763722339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6833","nonce":1763722339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6831","nonce":1763722339}]},"public_addr":"172.21.15.176:6827/1763722339","cluster_addr":"172.21.15.176:6829/1763722339","heartbeat_back_addr":"172.21.15.176:6833/1763722339","heartbeat_front_addr":"172.21.15.176:6831/1763722339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:23.124937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:22.444308+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:25.014164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:32.163410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T22:57:42.128 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T22:57:42.128 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd dump --format=json 2022-03-08T22:57:42.638 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:42.638 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":20,"fsid":"b6225762-85ac-4d55-aa25-1c146868c4f1","created":"2022-03-08T22:55:16.768945+0000","modified":"2022-03-08T22:57:34.749672+0000","last_up_change":"2022-03-08T22:57:33.640261+0000","last_in_change":"2022-03-08T22:57:00.615232+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T22:57:29.218823+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8f69b98f-f349-4192-b4f4-4d8f51a04951","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6811","nonce":3750486074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6813","nonce":3750486074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6817","nonce":3750486074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":3750486074},{"type":"v1","addr":"172.21.15.176:6815","nonce":3750486074}]},"public_addr":"172.21.15.176:6811/3750486074","cluster_addr":"172.21.15.176:6813/3750486074","heartbeat_back_addr":"172.21.15.176:6817/3750486074","heartbeat_front_addr":"172.21.15.176:6815/3750486074","state":["exists","up"]},{"osd":1,"uuid":"3c1dc57b-3e37-4862-9d1e-8185d624d2c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6803","nonce":2786447556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6805","nonce":2786447556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6809","nonce":2786447556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":2786447556},{"type":"v1","addr":"172.21.15.176:6807","nonce":2786447556}]},"public_addr":"172.21.15.176:6803/2786447556","cluster_addr":"172.21.15.176:6805/2786447556","heartbeat_back_addr":"172.21.15.176:6809/2786447556","heartbeat_front_addr":"172.21.15.176:6807/2786447556","state":["exists","up"]},{"osd":2,"uuid":"20e74f9b-ac2e-4200-8196-38a3b780c9b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6819","nonce":3585544879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6821","nonce":3585544879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6825","nonce":3585544879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":3585544879},{"type":"v1","addr":"172.21.15.176:6823","nonce":3585544879}]},"public_addr":"172.21.15.176:6819/3585544879","cluster_addr":"172.21.15.176:6821/3585544879","heartbeat_back_addr":"172.21.15.176:6825/3585544879","heartbeat_front_addr":"172.21.15.176:6823/3585544879","state":["exists","up"]},{"osd":3,"uuid":"2db5663d-d81d-4a8e-b31d-b1812c53d68a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6827","nonce":1763722339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6829","nonce":1763722339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6833","nonce":1763722339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1763722339},{"type":"v1","addr":"172.21.15.176:6831","nonce":1763722339}]},"public_addr":"172.21.15.176:6827/1763722339","cluster_addr":"172.21.15.176:6829/1763722339","heartbeat_back_addr":"172.21.15.176:6833/1763722339","heartbeat_front_addr":"172.21.15.176:6831/1763722339","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:23.124937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:22.444308+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:25.014164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T22:57:32.163410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T22:57:42.656 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph tell osd.0 flush_pg_stats 2022-03-08T22:57:42.657 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph tell osd.1 flush_pg_stats 2022-03-08T22:57:42.657 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph tell osd.2 flush_pg_stats 2022-03-08T22:57:42.658 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph tell osd.3 flush_pg_stats 2022-03-08T22:57:42.973 INFO:teuthology.orchestra.run.smithi176.stdout:81604378628 2022-03-08T22:57:42.973 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.3 2022-03-08T22:57:43.075 INFO:teuthology.orchestra.run.smithi176.stdout:51539607557 2022-03-08T22:57:43.076 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.0 2022-03-08T22:57:43.137 INFO:teuthology.orchestra.run.smithi176.stdout:51539607557 2022-03-08T22:57:43.138 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.1 2022-03-08T22:57:43.178 INFO:teuthology.orchestra.run.smithi176.stdout:60129542150 2022-03-08T22:57:43.178 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.2 2022-03-08T22:57:43.605 INFO:teuthology.orchestra.run.smithi176.stdout:81604378627 2022-03-08T22:57:43.631 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378628 got 81604378627 for osd.3 2022-03-08T22:57:43.645 INFO:teuthology.orchestra.run.smithi176.stdout:51539607556 2022-03-08T22:57:43.667 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.0 2022-03-08T22:57:43.731 INFO:teuthology.orchestra.run.smithi176.stdout:51539607556 2022-03-08T22:57:43.751 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.1 2022-03-08T22:57:43.756 INFO:teuthology.orchestra.run.smithi176.stdout:60129542148 2022-03-08T22:57:43.776 INFO:tasks.rook.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.2 2022-03-08T22:57:44.632 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.3 2022-03-08T22:57:44.669 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.0 2022-03-08T22:57:44.752 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.1 2022-03-08T22:57:44.777 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.2 2022-03-08T22:57:45.255 INFO:teuthology.orchestra.run.smithi176.stdout:51539607556 2022-03-08T22:57:45.288 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.1 2022-03-08T22:57:45.334 INFO:teuthology.orchestra.run.smithi176.stdout:81604378627 2022-03-08T22:57:45.352 INFO:teuthology.orchestra.run.smithi176.stdout:51539607556 2022-03-08T22:57:45.353 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378628 got 81604378627 for osd.3 2022-03-08T22:57:45.354 INFO:teuthology.orchestra.run.smithi176.stdout:60129542148 2022-03-08T22:57:45.372 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.0 2022-03-08T22:57:45.375 INFO:tasks.rook.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.2 2022-03-08T22:57:46.289 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.1 2022-03-08T22:57:46.355 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.3 2022-03-08T22:57:46.373 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.0 2022-03-08T22:57:46.376 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd last-stat-seq osd.2 2022-03-08T22:57:46.842 INFO:teuthology.orchestra.run.smithi176.stdout:51539607558 2022-03-08T22:57:46.862 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607557 got 51539607558 for osd.1 2022-03-08T22:57:46.862 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:46.951 INFO:teuthology.orchestra.run.smithi176.stdout:60129542150 2022-03-08T22:57:46.969 INFO:tasks.rook.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2022-03-08T22:57:46.970 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:46.988 INFO:teuthology.orchestra.run.smithi176.stdout:81604378629 2022-03-08T22:57:47.001 INFO:teuthology.orchestra.run.smithi176.stdout:51539607558 2022-03-08T22:57:47.005 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378628 got 81604378629 for osd.3 2022-03-08T22:57:47.006 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:47.022 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607557 got 51539607558 for osd.0 2022-03-08T22:57:47.022 DEBUG:teuthology.parallel:result is None 2022-03-08T22:57:47.023 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T22:57:47.024 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph pg dump --format=json 2022-03-08T22:57:47.513 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:47.516 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2022-03-08T22:57:47.530 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2022-03-08T22:57:46.739425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23204,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":20736,"kb_avail":374957404,"statfs":{"total":383980142592,"available":383956381696,"internally_reserved":0,"allocated":2101248,"data_stored":1579816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21233664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002718"},"pg_stats":[{"pgid":"1.0","version":"20'97","reported_seq":140,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-03-08T22:57:35.220190+0000","last_change":"2022-03-08T22:57:31.218023+0000","last_active":"2022-03-08T22:57:35.220190+0000","last_peered":"2022-03-08T22:57:35.220190+0000","last_clean":"2022-03-08T22:57:35.220190+0000","last_became_active":"2022-03-08T22:57:31.216868+0000","last_became_peered":"2022-03-08T22:57:31.216868+0000","last_unstale":"2022-03-08T22:57:35.220190+0000","last_undegraded":"2022-03-08T22:57:35.220190+0000","last_fullsized":"2022-03-08T22:57:35.220190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T22:57:29.218956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T22:57:29.218956+0000","last_clean_scrub_stamp":"2022-03-08T22:57:29.218956+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T06:23:09.694547+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5040,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740112,"statfs":{"total":95995035648,"available":95989874688,"internally_reserved":0,"allocated":172032,"data_stored":44737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6012,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739140,"statfs":{"total":95995035648,"available":95988879360,"internally_reserved":0,"allocated":643072,"data_stored":511693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":643072,"data_stored":511693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":643072,"data_stored":511693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T22:57:47.531 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph pg dump --format=json 2022-03-08T22:57:48.039 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:48.042 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2022-03-08T22:57:48.060 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2022-03-08T22:57:46.739425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23204,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":20736,"kb_avail":374957404,"statfs":{"total":383980142592,"available":383956381696,"internally_reserved":0,"allocated":2101248,"data_stored":1579816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21233664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002718"},"pg_stats":[{"pgid":"1.0","version":"20'97","reported_seq":140,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-03-08T22:57:35.220190+0000","last_change":"2022-03-08T22:57:31.218023+0000","last_active":"2022-03-08T22:57:35.220190+0000","last_peered":"2022-03-08T22:57:35.220190+0000","last_clean":"2022-03-08T22:57:35.220190+0000","last_became_active":"2022-03-08T22:57:31.216868+0000","last_became_peered":"2022-03-08T22:57:31.216868+0000","last_unstale":"2022-03-08T22:57:35.220190+0000","last_undegraded":"2022-03-08T22:57:35.220190+0000","last_fullsized":"2022-03-08T22:57:35.220190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T22:57:29.218956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T22:57:29.218956+0000","last_clean_scrub_stamp":"2022-03-08T22:57:29.218956+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T06:23:09.694547+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5040,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740112,"statfs":{"total":95995035648,"available":95989874688,"internally_reserved":0,"allocated":172032,"data_stored":44737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6012,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739140,"statfs":{"total":95995035648,"available":95988879360,"internally_reserved":0,"allocated":643072,"data_stored":511693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":643072,"data_stored":511693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":643072,"data_stored":511693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T22:57:48.061 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T22:57:48.061 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T22:57:48.061 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T22:57:48.062 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph health --format=json 2022-03-08T22:57:48.646 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:48.646 INFO:teuthology.orchestra.run.smithi176.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T22:57:48.664 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T22:57:48.665 INFO:tasks.rook:Rook complete, yielding 2022-03-08T22:57:48.665 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T22:57:48.679 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph -s 2022-03-08T22:57:49.299 INFO:teuthology.orchestra.run.smithi176.stdout: cluster: 2022-03-08T22:57:49.300 INFO:teuthology.orchestra.run.smithi176.stdout: id: b6225762-85ac-4d55-aa25-1c146868c4f1 2022-03-08T22:57:49.300 INFO:teuthology.orchestra.run.smithi176.stdout: health: HEALTH_OK 2022-03-08T22:57:49.300 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:49.300 INFO:teuthology.orchestra.run.smithi176.stdout: services: 2022-03-08T22:57:49.301 INFO:teuthology.orchestra.run.smithi176.stdout: mon: 1 daemons, quorum a (age 2m) 2022-03-08T22:57:49.301 INFO:teuthology.orchestra.run.smithi176.stdout: mgr: a(active, since 117s) 2022-03-08T22:57:49.301 INFO:teuthology.orchestra.run.smithi176.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 48s) 2022-03-08T22:57:49.302 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:49.302 INFO:teuthology.orchestra.run.smithi176.stdout: data: 2022-03-08T22:57:49.302 INFO:teuthology.orchestra.run.smithi176.stdout: pools: 1 pools, 1 pgs 2022-03-08T22:57:49.302 INFO:teuthology.orchestra.run.smithi176.stdout: objects: 2 objects, 577 KiB 2022-03-08T22:57:49.303 INFO:teuthology.orchestra.run.smithi176.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-03-08T22:57:49.303 INFO:teuthology.orchestra.run.smithi176.stdout: pgs: 1 active+clean 2022-03-08T22:57:49.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T22:57:49.323 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status 2022-03-08T22:57:49.919 INFO:teuthology.orchestra.run.smithi176.stdout:Backend: rook 2022-03-08T22:57:49.919 INFO:teuthology.orchestra.run.smithi176.stdout:Available: Yes 2022-03-08T22:57:49.941 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ps 2022-03-08T22:57:50.479 INFO:teuthology.orchestra.run.smithi176.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T22:57:50.480 INFO:teuthology.orchestra.run.smithi176.stdout:crashcollector.crash smithi176 running (62s) 0s ago 62s - - 1c5288aef625 2022-03-08T22:57:50.480 INFO:teuthology.orchestra.run.smithi176.stdout:mgr.a smithi176 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-08T22:57:50.480 INFO:teuthology.orchestra.run.smithi176.stdout:mon.a smithi176 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-08T22:57:50.480 INFO:teuthology.orchestra.run.smithi176.stdout:osd.0 smithi176 running (62s) 0s ago 63s - - 1c5288aef625 2022-03-08T22:57:50.481 INFO:teuthology.orchestra.run.smithi176.stdout:osd.1 smithi176 running (60s) 0s ago 60s - - 1c5288aef625 2022-03-08T22:57:50.481 INFO:teuthology.orchestra.run.smithi176.stdout:osd.2 smithi176 running (58s) 0s ago 59s - - 1c5288aef625 2022-03-08T22:57:50.481 INFO:teuthology.orchestra.run.smithi176.stdout:osd.3 smithi176 running (43s) 0s ago 44s - - 1c5288aef625 2022-03-08T22:57:50.499 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls 2022-03-08T22:57:51.095 INFO:teuthology.orchestra.run.smithi176.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T22:57:51.095 INFO:teuthology.orchestra.run.smithi176.stdout:crash 1/1 0s ago 63s * 2022-03-08T22:57:51.095 INFO:teuthology.orchestra.run.smithi176.stdout:mgr 1/1 0s ago 2m count:1 2022-03-08T22:57:51.096 INFO:teuthology.orchestra.run.smithi176.stdout:mon 1/1 0s ago 2m count:1 2022-03-08T22:57:51.096 INFO:teuthology.orchestra.run.smithi176.stdout:osd 8 0s ago 64s 2022-03-08T22:57:51.096 INFO:teuthology.orchestra.run.smithi176.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T22:57:51.118 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch device ls 2022-03-08T22:57:51.638 INFO:teuthology.orchestra.run.smithi176.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T22:57:51.639 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T22:57:51.639 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T22:57:51.639 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T22:57:51.639 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T22:57:51.661 INFO:teuthology.run_tasks:Running task install... 2022-03-08T22:57:51.671 DEBUG:teuthology.task.install:project ceph 2022-03-08T22:57:51.672 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4'}} 2022-03-08T22:57:51.672 DEBUG:teuthology.task.install:config {'host.a': None, 'flavor': 'default', 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4'} 2022-03-08T22:57:51.672 INFO:teuthology.task.install:Using flavor: default 2022-03-08T22:57:51.681 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-08T22:57:51.681 INFO:teuthology.task.install:extra packages: [] 2022-03-08T22:57:51.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-03-08T22:57:51.725 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-03-08T22:57:51.729 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:57:51.730 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-key list | grep Ceph 2022-03-08T22:57:51.876 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-08T22:57:51.934 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-08T22:57:51.934 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph.com (release key) 2022-03-08T22:57:51.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-03-08T22:57:51.986 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-03-08T22:57:51.992 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:57:51.992 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-08T22:57:51.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:57:52.210 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default/ 2022-03-08T22:57:52.210 INFO:teuthology.task.install.deb:Package version is 17.0.0-11002-g94cb3430-1focal 2022-03-08T22:57:52.345 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:57:52.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-08T22:57:52.357 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2022-03-08T22:57:52.556 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:57:52.626 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal InRelease 2022-03-08T22:57:52.649 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-08T22:57:52.670 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:57:52.683 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T22:57:52.752 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:57:52.804 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal Release.gpg 2022-03-08T22:57:52.827 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main i386 Packages [3,353 B] 2022-03-08T22:57:52.837 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:57:52.860 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2022-03-08T22:57:53.208 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 37.7 kB in 1s (62.1 kB/s) 2022-03-08T22:57:54.155 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:57:54.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11002-g94cb3430-1focal cephadm=17.0.0-11002-g94cb3430-1focal ceph-mds=17.0.0-11002-g94cb3430-1focal ceph-mgr=17.0.0-11002-g94cb3430-1focal ceph-common=17.0.0-11002-g94cb3430-1focal ceph-fuse=17.0.0-11002-g94cb3430-1focal ceph-test=17.0.0-11002-g94cb3430-1focal ceph-volume=17.0.0-11002-g94cb3430-1focal radosgw=17.0.0-11002-g94cb3430-1focal python3-rados=17.0.0-11002-g94cb3430-1focal python3-rgw=17.0.0-11002-g94cb3430-1focal python3-cephfs=17.0.0-11002-g94cb3430-1focal python3-rbd=17.0.0-11002-g94cb3430-1focal libcephfs2=17.0.0-11002-g94cb3430-1focal libcephfs-dev=17.0.0-11002-g94cb3430-1focal librados2=17.0.0-11002-g94cb3430-1focal librbd1=17.0.0-11002-g94cb3430-1focal rbd-fuse=17.0.0-11002-g94cb3430-1focal 2022-03-08T22:57:54.255 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2022-03-08T22:57:54.451 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2022-03-08T22:57:54.451 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2022-03-08T22:57:54.625 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2022-03-08T22:57:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-08T22:57:54.627 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-08T22:57:54.627 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2022-03-08T22:57:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-08T22:57:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2022-03-08T22:57:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T22:57:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T22:57:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T22:57:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T22:57:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-03-08T22:57:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T22:57:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T22:57:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T22:57:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T22:57:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T22:57:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T22:57:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-08T22:57:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-08T22:57:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-08T22:57:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-08T22:57:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-08T22:57:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket python3-webtest python3-werkzeug xmlstarlet 2022-03-08T22:57:54.633 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2022-03-08T22:57:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-08T22:57:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-08T22:57:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-08T22:57:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-08T22:57:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-08T22:57:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-08T22:57:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-08T22:57:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-08T22:57:54.635 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2022-03-08T22:57:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-08T22:57:54.742 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2022-03-08T22:57:54.743 INFO:teuthology.orchestra.run.smithi176.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-08T22:57:54.743 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-08T22:57:54.745 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-08T22:57:54.745 INFO:teuthology.orchestra.run.smithi176.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-08T22:57:54.746 INFO:teuthology.orchestra.run.smithi176.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-08T22:57:54.746 INFO:teuthology.orchestra.run.smithi176.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-08T22:57:54.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-08T22:57:54.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-03-08T22:57:54.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-08T22:57:54.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-08T22:57:54.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-08T22:57:54.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-08T22:57:54.747 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-08T22:57:54.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-08T22:57:54.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-08T22:57:54.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-08T22:57:54.748 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-08T22:57:54.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-08T22:57:54.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-08T22:57:54.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-08T22:57:54.749 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse xmlstarlet 2022-03-08T22:57:54.749 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2022-03-08T22:57:54.750 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 librbd1 2022-03-08T22:57:54.900 INFO:teuthology.orchestra.run.smithi176.stdout:2 upgraded, 95 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T22:57:54.900 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 181 MB of archives. 2022-03-08T22:57:54.900 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 845 MB of additional disk space will be used. 2022-03-08T22:57:54.901 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11002-g94cb3430-1focal [3,484 kB] 2022-03-08T22:57:54.937 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-08T22:57:55.126 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11002-g94cb3430-1focal [3,534 kB] 2022-03-08T22:57:55.230 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11002-g94cb3430-1focal [756 kB] 2022-03-08T22:57:55.255 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11002-g94cb3430-1focal [412 kB] 2022-03-08T22:57:55.269 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11002-g94cb3430-1focal [22.4 kB] 2022-03-08T22:57:55.269 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11002-g94cb3430-1focal [200 kB] 2022-03-08T22:57:55.276 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11002-g94cb3430-1focal [41.9 kB] 2022-03-08T22:57:55.277 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11002-g94cb3430-1focal [386 kB] 2022-03-08T22:57:55.290 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11002-g94cb3430-1focal [4,162 kB] 2022-03-08T22:57:55.302 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-08T22:57:55.310 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-08T22:57:55.330 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-08T22:57:55.384 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-08T22:57:55.427 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11002-g94cb3430-1focal [116 kB] 2022-03-08T22:57:55.430 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11002-g94cb3430-1focal [478 kB] 2022-03-08T22:57:55.445 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11002-g94cb3430-1focal [24.4 MB] 2022-03-08T22:57:55.517 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-08T22:57:55.524 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-08T22:57:55.578 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-08T22:57:55.584 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-08T22:57:55.588 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-08T22:57:55.607 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-08T22:57:55.610 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-08T22:57:55.612 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-08T22:57:55.613 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-08T22:57:55.644 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-08T22:57:55.645 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-08T22:57:55.757 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-08T22:57:55.763 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-08T22:57:55.764 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-08T22:57:55.771 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-08T22:57:55.777 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-08T22:57:55.778 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-08T22:57:55.898 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-08T22:57:55.899 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-08T22:57:55.899 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-08T22:57:55.900 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-08T22:57:55.948 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-08T22:57:55.958 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-08T22:57:55.958 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-08T22:57:55.961 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-08T22:57:55.983 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-08T22:57:56.042 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-08T22:57:56.043 INFO:teuthology.orchestra.run.smithi176.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-08T22:57:56.044 INFO:teuthology.orchestra.run.smithi176.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-08T22:57:56.344 INFO:teuthology.orchestra.run.smithi176.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11002-g94cb3430-1focal [6,432 kB] 2022-03-08T22:57:56.434 INFO:teuthology.orchestra.run.smithi176.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-08T22:57:56.461 INFO:teuthology.orchestra.run.smithi176.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-08T22:57:56.463 INFO:teuthology.orchestra.run.smithi176.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-08T22:57:56.521 INFO:teuthology.orchestra.run.smithi176.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11002-g94cb3430-1focal [211 kB] 2022-03-08T22:57:56.523 INFO:teuthology.orchestra.run.smithi176.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11002-g94cb3430-1focal [151 kB] 2022-03-08T22:57:56.524 INFO:teuthology.orchestra.run.smithi176.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11002-g94cb3430-1focal [1,575 kB] 2022-03-08T22:57:56.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-08T22:57:56.614 INFO:teuthology.orchestra.run.smithi176.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-08T22:57:56.615 INFO:teuthology.orchestra.run.smithi176.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11002-g94cb3430-1focal [7,812 kB] 2022-03-08T22:57:56.615 INFO:teuthology.orchestra.run.smithi176.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-08T22:57:56.616 INFO:teuthology.orchestra.run.smithi176.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-08T22:57:56.616 INFO:teuthology.orchestra.run.smithi176.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-08T22:57:56.617 INFO:teuthology.orchestra.run.smithi176.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-08T22:57:56.617 INFO:teuthology.orchestra.run.smithi176.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-08T22:57:56.618 INFO:teuthology.orchestra.run.smithi176.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-08T22:57:56.618 INFO:teuthology.orchestra.run.smithi176.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-08T22:57:56.697 INFO:teuthology.orchestra.run.smithi176.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-08T22:57:56.701 INFO:teuthology.orchestra.run.smithi176.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-08T22:57:56.703 INFO:teuthology.orchestra.run.smithi176.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-08T22:57:56.705 INFO:teuthology.orchestra.run.smithi176.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-08T22:57:56.706 INFO:teuthology.orchestra.run.smithi176.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-08T22:57:56.708 INFO:teuthology.orchestra.run.smithi176.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-08T22:57:56.710 INFO:teuthology.orchestra.run.smithi176.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-08T22:57:56.714 INFO:teuthology.orchestra.run.smithi176.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-08T22:57:56.714 INFO:teuthology.orchestra.run.smithi176.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-08T22:57:56.778 INFO:teuthology.orchestra.run.smithi176.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-08T22:57:56.797 INFO:teuthology.orchestra.run.smithi176.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-08T22:57:56.799 INFO:teuthology.orchestra.run.smithi176.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-08T22:57:56.816 INFO:teuthology.orchestra.run.smithi176.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11002-g94cb3430-1focal [28.6 MB] 2022-03-08T22:57:56.828 INFO:teuthology.orchestra.run.smithi176.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-08T22:57:56.829 INFO:teuthology.orchestra.run.smithi176.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-08T22:57:56.829 INFO:teuthology.orchestra.run.smithi176.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-08T22:57:56.830 INFO:teuthology.orchestra.run.smithi176.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-08T22:57:56.831 INFO:teuthology.orchestra.run.smithi176.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-08T22:57:56.832 INFO:teuthology.orchestra.run.smithi176.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-08T22:57:56.861 INFO:teuthology.orchestra.run.smithi176.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-08T22:57:56.943 INFO:teuthology.orchestra.run.smithi176.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-08T22:57:56.944 INFO:teuthology.orchestra.run.smithi176.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-08T22:57:57.897 INFO:teuthology.orchestra.run.smithi176.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11002-g94cb3430-1focal [3,796 B] 2022-03-08T22:57:57.897 INFO:teuthology.orchestra.run.smithi176.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11002-g94cb3430-1focal [863 kB] 2022-03-08T22:57:57.903 INFO:teuthology.orchestra.run.smithi176.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11002-g94cb3430-1focal [2,397 kB] 2022-03-08T22:57:57.921 INFO:teuthology.orchestra.run.smithi176.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11002-g94cb3430-1focal [1,677 kB] 2022-03-08T22:57:57.980 INFO:teuthology.orchestra.run.smithi176.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11002-g94cb3430-1focal [7,888 kB] 2022-03-08T22:57:58.237 INFO:teuthology.orchestra.run.smithi176.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11002-g94cb3430-1focal [3,912 B] 2022-03-08T22:57:58.349 INFO:teuthology.orchestra.run.smithi176.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11002-g94cb3430-1focal [50.5 MB] 2022-03-08T22:58:00.119 INFO:teuthology.orchestra.run.smithi176.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11002-g94cb3430-1focal [219 kB] 2022-03-08T22:58:00.121 INFO:teuthology.orchestra.run.smithi176.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11002-g94cb3430-1focal [69.5 kB] 2022-03-08T22:58:00.121 INFO:teuthology.orchestra.run.smithi176.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11002-g94cb3430-1focal [21.7 kB] 2022-03-08T22:58:00.122 INFO:teuthology.orchestra.run.smithi176.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11002-g94cb3430-1focal [12.7 MB] 2022-03-08T22:58:00.474 INFO:teuthology.orchestra.run.smithi176.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-03-07-0958/94cb3430b74fb099b8927e3e6ad50dfd969173f4/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11002-g94cb3430-1focal [78.9 kB] 2022-03-08T22:58:01.112 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 181 MB in 6s (31.7 MB/s) 2022-03-08T22:58:01.290 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-08T22:58:01.343 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-08T22:58:01.347 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-08T22:58:01.464 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T22:58:01.885 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../01-librbd1_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:02.111 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librbd1 (17.0.0-11002-g94cb3430-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T22:58:02.830 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../02-librados2_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:03.010 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librados2 (17.0.0-11002-g94cb3430-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-08T22:58:03.789 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs2. 2022-03-08T22:58:03.799 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:03.847 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:04.319 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rados. 2022-03-08T22:58:04.333 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../04-python3-rados_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:04.377 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rados (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:04.822 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-08T22:58:04.835 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:04.880 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-argparse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:05.189 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cephfs. 2022-03-08T22:58:05.203 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:05.292 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cephfs (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:05.671 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-08T22:58:05.685 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:05.730 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:06.092 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-prettytable. 2022-03-08T22:58:06.103 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-03-08T22:58:06.141 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-08T22:58:06.511 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rbd. 2022-03-08T22:58:06.525 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:06.569 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rbd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:06.982 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-08T22:58:06.998 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-08T22:58:07.041 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T22:58:07.436 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-08T22:58:07.452 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-08T22:58:07.486 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T22:58:07.906 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-08T22:58:07.914 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-08T22:58:08.007 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T22:58:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librgw2. 2022-03-08T22:58:08.408 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../13-librgw2_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:08.494 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librgw2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:09.197 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rgw. 2022-03-08T22:58:09.211 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:09.246 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:09.711 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liboath0. 2022-03-08T22:58:09.727 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-08T22:58:09.851 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-08T22:58:10.188 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libradosstriper1. 2022-03-08T22:58:10.200 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../16-libradosstriper1_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:10.245 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libradosstriper1 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:10.749 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-common. 2022-03-08T22:58:10.760 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../17-ceph-common_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:10.799 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:13.628 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-base. 2022-03-08T22:58:13.640 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../18-ceph-base_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:13.686 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-base (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:14.824 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cryptography. 2022-03-08T22:58:14.839 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-08T22:58:14.925 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T22:58:15.287 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-openssl. 2022-03-08T22:58:15.299 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-08T22:58:15.337 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-08T22:58:15.615 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-08T22:58:15.626 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:15.690 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:16.192 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-08T22:58:16.203 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-08T22:58:16.239 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T22:58:16.577 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-08T22:58:16.590 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-08T22:58:16.631 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-08T22:58:17.095 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jwt. 2022-03-08T22:58:17.107 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-08T22:58:17.175 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T22:58:17.498 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-logutils. 2022-03-08T22:58:17.510 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-03-08T22:58:17.540 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-08T22:58:17.883 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-08T22:58:17.896 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-08T22:58:17.935 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-08T22:58:18.452 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-mako. 2022-03-08T22:58:18.464 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-08T22:58:18.527 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T22:58:18.889 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-08T22:58:18.902 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-08T22:58:18.949 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-08T22:58:19.260 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-08T22:58:19.274 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-08T22:58:19.318 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T22:58:19.676 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webob. 2022-03-08T22:58:19.689 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-08T22:58:19.795 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-08T22:58:20.222 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-08T22:58:20.231 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-08T22:58:20.288 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T22:58:20.701 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bs4. 2022-03-08T22:58:20.713 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-03-08T22:58:20.752 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-08T22:58:21.153 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-waitress. 2022-03-08T22:58:21.166 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1_all.deb ... 2022-03-08T22:58:21.216 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-08T22:58:21.570 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempita. 2022-03-08T22:58:21.585 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-03-08T22:58:21.621 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-08T22:58:22.050 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-paste. 2022-03-08T22:58:22.060 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-08T22:58:22.130 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T22:58:22.582 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-08T22:58:22.593 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-08T22:58:22.640 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T22:58:22.938 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-08T22:58:22.951 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-08T22:58:22.996 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-08T22:58:23.457 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webtest. 2022-03-08T22:58:23.470 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-03-08T22:58:23.514 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-08T22:58:24.210 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pecan. 2022-03-08T22:58:24.225 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-08T22:58:24.269 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T22:58:24.731 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-08T22:58:24.744 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-08T22:58:24.789 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T22:58:25.233 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-08T22:58:25.247 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../41-libsqlite3-mod-ceph_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:25.292 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:25.703 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr. 2022-03-08T22:58:25.719 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../42-ceph-mgr_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:25.762 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:26.427 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mon. 2022-03-08T22:58:26.437 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../43-ceph-mon_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:26.521 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mon (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:27.762 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-osd. 2022-03-08T22:58:27.775 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../44-ceph-osd_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:27.821 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-osd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:31.440 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph. 2022-03-08T22:58:31.453 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../45-ceph_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:31.498 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:31.959 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-fuse. 2022-03-08T22:58:31.975 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../46-ceph-fuse_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:32.017 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:32.554 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mds. 2022-03-08T22:58:32.566 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../47-ceph-mds_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:32.604 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mds (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:33.160 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-08T22:58:33.174 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2022-03-08T22:58:33.209 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-08T22:58:33.467 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjs-underscore. 2022-03-08T22:58:33.481 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../49-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-08T22:58:33.525 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T22:58:33.807 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-08T22:58:33.822 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../50-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-08T22:58:33.867 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T22:58:34.425 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-routes. 2022-03-08T22:58:34.435 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../51-python3-routes_2.4.1-2_all.deb ... 2022-03-08T22:58:34.469 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-08T22:58:34.847 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-08T22:58:34.861 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../52-ceph-mgr-dashboard_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:34.913 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:35.568 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-decorator. 2022-03-08T22:58:35.577 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../53-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-08T22:58:35.652 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T22:58:36.005 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-08T22:58:36.020 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../54-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-08T22:58:36.058 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T22:58:36.529 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-scipy. 2022-03-08T22:58:36.544 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../55-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-08T22:58:36.630 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-08T22:58:38.466 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-08T22:58:38.479 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../56-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-08T22:58:38.534 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T22:58:39.225 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-joblib. 2022-03-08T22:58:39.237 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../57-python3-joblib_0.14.0-3_all.deb ... 2022-03-08T22:58:39.277 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-08T22:58:39.747 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn. 2022-03-08T22:58:39.756 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../58-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-08T22:58:39.807 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T22:58:40.903 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-08T22:58:40.915 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../59-ceph-mgr-diskprediction-local_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:40.952 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:42.209 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cachetools. 2022-03-08T22:58:42.223 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../60-python3-cachetools_4.0.0-1_all.deb ... 2022-03-08T22:58:42.278 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-08T22:58:42.738 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-08T22:58:42.752 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../61-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-08T22:58:42.805 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T22:58:43.224 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-08T22:58:43.236 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../62-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-08T22:58:43.300 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T22:58:43.669 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rsa. 2022-03-08T22:58:43.684 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../63-python3-rsa_4.0-3_all.deb ... 2022-03-08T22:58:43.728 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-08T22:58:44.160 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-google-auth. 2022-03-08T22:58:44.169 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../64-python3-google-auth_1.5.1-2_all.deb ... 2022-03-08T22:58:44.226 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-08T22:58:44.613 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-blinker. 2022-03-08T22:58:44.622 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../65-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-08T22:58:44.662 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T22:58:44.982 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-08T22:58:44.995 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../66-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-08T22:58:45.040 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T22:58:45.508 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-08T22:58:45.522 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../67-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-08T22:58:45.563 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T22:58:45.997 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-websocket. 2022-03-08T22:58:46.011 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../68-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-08T22:58:46.055 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T22:58:46.428 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-08T22:58:46.437 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../69-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-08T22:58:46.478 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T22:58:47.216 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-08T22:58:47.226 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../70-ceph-mgr-k8sevents_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:47.274 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:47.661 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-08T22:58:47.672 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../71-libonig5_6.9.4-1_amd64.deb ... 2022-03-08T22:58:47.734 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-08T22:58:48.176 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-08T22:58:48.187 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../72-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T22:58:48.238 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:58:48.553 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package jq. 2022-03-08T22:58:48.566 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../73-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-08T22:58:48.620 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:58:49.062 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-08T22:58:49.077 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../74-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-08T22:58:49.121 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T22:58:49.497 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package xmlstarlet. 2022-03-08T22:58:49.512 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../75-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-08T22:58:49.557 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-08T22:58:50.023 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-test. 2022-03-08T22:58:50.038 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../76-ceph-test_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:50.145 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-test (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:55.015 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-volume. 2022-03-08T22:58:55.029 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../77-ceph-volume_17.0.0-11002-g94cb3430-1focal_all.deb ... 2022-03-08T22:58:55.068 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-volume (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:55.577 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephadm. 2022-03-08T22:58:55.592 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../78-cephadm_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:55.634 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephadm (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:56.014 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-08T22:58:56.027 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../79-libcephfs-dev_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:58:56.204 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs-dev (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:58:56.697 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package nvme-cli. 2022-03-08T22:58:56.707 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../80-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-08T22:58:56.762 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T22:58:57.322 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-08T22:58:57.335 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../81-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-08T22:58:57.380 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T22:58:57.716 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-attr. 2022-03-08T22:58:57.727 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../82-python3-attr_19.3.0-2_all.deb ... 2022-03-08T22:58:57.797 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-08T22:58:58.284 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webencodings. 2022-03-08T22:58:58.297 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../83-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-08T22:58:58.342 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T22:58:58.657 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-html5lib. 2022-03-08T22:58:58.666 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../84-python3-html5lib_1.0.1-2_all.deb ... 2022-03-08T22:58:58.710 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-08T22:58:59.114 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-08T22:58:59.127 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../85-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-08T22:58:59.205 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T22:58:59.741 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-packaging. 2022-03-08T22:58:59.755 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../86-python3-packaging_20.3-1_all.deb ... 2022-03-08T22:58:59.799 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-08T22:59:00.183 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastescript. 2022-03-08T22:59:00.200 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../87-python3-pastescript_2.0.2-4_all.deb ... 2022-03-08T22:59:00.241 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-08T22:59:00.632 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pluggy. 2022-03-08T22:59:00.642 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../88-python3-pluggy_0.13.0-2_all.deb ... 2022-03-08T22:59:00.682 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-08T22:59:01.556 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-psutil. 2022-03-08T22:59:01.565 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../89-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-08T22:59:01.969 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T22:59:02.502 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-py. 2022-03-08T22:59:02.513 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../90-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-08T22:59:02.556 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T22:59:02.917 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-08T22:59:02.932 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../91-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-08T22:59:02.992 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T22:59:03.446 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-08T22:59:03.454 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../92-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-08T22:59:03.495 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T22:59:03.799 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pytest. 2022-03-08T22:59:03.812 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../93-python3-pytest_4.6.9-1_all.deb ... 2022-03-08T22:59:03.856 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-08T22:59:04.198 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplejson. 2022-03-08T22:59:04.212 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../94-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-08T22:59:04.290 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T22:59:04.844 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package radosgw. 2022-03-08T22:59:04.855 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../95-radosgw_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:04.894 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking radosgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:06.369 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package rbd-fuse. 2022-03-08T22:59:06.381 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../96-rbd-fuse_17.0.0-11002-g94cb3430-1focal_amd64.deb ... 2022-03-08T22:59:06.448 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking rbd-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:06.826 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-08T22:59:07.202 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-08T22:59:07.329 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-08T22:59:07.514 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephadm (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:08.201 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user cephadm....done 2022-03-08T22:59:08.400 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-08T22:59:08.760 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-08T22:59:09.103 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-08T22:59:09.229 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-08T22:59:09.557 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-08T22:59:09.980 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-08T22:59:10.266 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-08T22:59:10.592 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-08T22:59:10.893 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-08T22:59:11.288 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-argparse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:11.627 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-08T22:59:11.761 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-08T22:59:12.037 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-08T22:59:12.355 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-08T22:59:12.707 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-08T22:59:12.996 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-08T22:59:13.364 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-08T22:59:13.641 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-08T22:59:13.944 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-08T22:59:14.303 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-08T22:59:14.429 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-08T22:59:14.757 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-08T22:59:15.170 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-08T22:59:15.522 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-08T22:59:15.909 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:16.227 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-08T22:59:16.584 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-08T22:59:16.768 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-08T22:59:17.140 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-08T22:59:17.283 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-08T22:59:17.409 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-08T22:59:17.719 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-08T22:59:18.038 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-08T22:59:18.336 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-08T22:59:18.479 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-08T22:59:18.813 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-08T22:59:19.148 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-08T22:59:19.450 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-08T22:59:19.540 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-08T22:59:19.809 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-08T22:59:19.951 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-08T22:59:20.296 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-08T22:59:20.631 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-08T22:59:20.779 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-08T22:59:20.981 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-08T22:59:21.408 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-08T22:59:21.698 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librados2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:21.833 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-08T22:59:23.496 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librgw2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:23.664 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:23.790 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:59:23.950 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-08T22:59:24.402 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-08T22:59:24.725 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-08T22:59:25.044 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-08T22:59:25.395 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs2 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:25.538 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-08T22:59:25.916 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libradosstriper1 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:26.059 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-08T22:59:26.383 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-08T22:59:27.649 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librbd1 (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:27.811 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-08T22:59:28.053 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-08T22:59:28.175 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-08T22:59:28.477 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-08T22:59:28.603 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-08T22:59:28.996 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-08T22:59:29.373 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-modules-core (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:29.499 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:29.727 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T22:59:29.728 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-08T22:59:30.691 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs-dev (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:30.817 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-08T22:59:31.169 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-08T22:59:31.593 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-08T22:59:31.954 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rados (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:32.088 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rbd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:32.239 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up rbd-fuse (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:32.367 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-08T22:59:32.664 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-08T22:59:33.795 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:33.938 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cephfs (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:34.106 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-08T22:59:35.217 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-common (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:35.929 INFO:teuthology.orchestra.run.smithi176.stdout:Adding group ceph....done 2022-03-08T22:59:36.279 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user ceph....done 2022-03-08T22:59:36.468 INFO:teuthology.orchestra.run.smithi176.stdout:Setting system user ceph properties....done 2022-03-08T22:59:36.490 INFO:teuthology.orchestra.run.smithi176.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-08T22:59:36.634 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-08T22:59:37.183 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-08T22:59:38.152 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-test (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:38.278 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up radosgw (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:38.570 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T22:59:38.570 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-08T22:59:39.930 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-base (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:40.321 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-08T22:59:41.258 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mds (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:41.486 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T22:59:41.487 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-08T22:59:42.400 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:42.630 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T22:59:42.631 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-08T22:59:43.560 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-osd (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:44.617 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T22:59:44.618 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-08T22:59:45.480 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:45.640 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:45.791 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mon (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:46.141 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T22:59:46.142 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-08T22:59:47.017 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:47.140 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-dashboard (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:47.293 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-volume (17.0.0-11002-g94cb3430-1focal) ... 2022-03-08T22:59:47.410 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-08T22:59:48.024 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:59:50.823 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:59:51.303 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-08T22:59:51.307 DEBUG:teuthology.parallel:result is None 2022-03-08T22:59:51.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo lsb_release -is 2022-03-08T22:59:51.350 INFO:teuthology.orchestra.run.smithi176.stdout:Ubuntu 2022-03-08T22:59:51.355 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T22:59:51.355 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-08T22:59:51.522 DEBUG:teuthology.orchestra.run.smithi176:> dpkg-query -W -f '${Version}' ceph 2022-03-08T22:59:51.546 INFO:teuthology.orchestra.run.smithi176.stdout:17.0.0-11002-g94cb3430-1focal 2022-03-08T22:59:51.547 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11002-g94cb3430-1focal 2022-03-08T22:59:51.547 INFO:teuthology.task.install:The correct ceph version 17.0.0-11002-g94cb3430-1focal is installed. 2022-03-08T22:59:51.548 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-08T22:59:51.548 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:59:51.549 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-08T22:59:51.561 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-08T22:59:51.562 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:59:51.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2022-03-08T22:59:51.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-08T22:59:51.632 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-08T22:59:51.633 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2022-03-08T22:59:51.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-08T22:59:51.648 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-08T22:59:51.703 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-08T22:59:51.718 INFO:tasks.radosbench:Beginning radosbench... 2022-03-08T22:59:51.718 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-08T22:59:51.719 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd pool create unique_pool_0 16 2022-03-08T22:59:52.876 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'unique_pool_0' created 2022-03-08T22:59:52.896 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-08T22:59:53.940 INFO:teuthology.orchestra.run.smithi176.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-08T22:59:54.963 DEBUG:teuthology.orchestra.run.smithi176:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-08T22:59:54.966 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T22:59:54.979 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch host label add '`hostname`' 'foo 2022-03-08T22:59:54.980 DEBUG:teuthology.orchestra.run.smithi176:> ceph' orch host ls '|' grep 'foo 2022-03-08T22:59:54.980 DEBUG:teuthology.orchestra.run.smithi176:> ceph' orch host label rm '`hostname`' 'foo 2022-03-08T22:59:54.980 DEBUG:teuthology.orchestra.run.smithi176:> ceph' orch host ls '|' grep -v 'foo 2022-03-08T22:59:54.981 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-03-08T22:59:55.073 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:hints = 1 2022-03-08T22:59:55.073 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-08T22:59:55.074 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Object prefix: benchmark_data_smithi176_46003 2022-03-08T22:59:55.074 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T22:59:55.075 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 0 0 0 0 0 0 - 0 2022-03-08T22:59:55.343 INFO:teuthology.orchestra.run.smithi176.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-08T22:59:55.375 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T22:59:55.388 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T22:59:55.388 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi176.front.sepia.ceph.com 2022-03-08T22:59:55.389 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T22:59:55.389 DEBUG:teuthology.orchestra.run.smithi176:> toolbox() { 2022-03-08T22:59:55.389 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T22:59:55.390 DEBUG:teuthology.orchestra.run.smithi176:> } 2022-03-08T22:59:55.390 DEBUG:teuthology.orchestra.run.smithi176:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T22:59:55.390 DEBUG:teuthology.orchestra.run.smithi176:> toolbox ceph orch osd rm 0 --force 2022-03-08T22:59:55.391 DEBUG:teuthology.orchestra.run.smithi176:> removed_pv="" 2022-03-08T22:59:55.391 DEBUG:teuthology.orchestra.run.smithi176:> while [ "$removed_pv" = "" ] 2022-03-08T22:59:55.391 DEBUG:teuthology.orchestra.run.smithi176:> do 2022-03-08T22:59:55.392 DEBUG:teuthology.orchestra.run.smithi176:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T22:59:55.392 DEBUG:teuthology.orchestra.run.smithi176:> sleep 3s 2022-03-08T22:59:55.392 DEBUG:teuthology.orchestra.run.smithi176:> done 2022-03-08T22:59:55.393 DEBUG:teuthology.orchestra.run.smithi176:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T22:59:55.393 DEBUG:teuthology.orchestra.run.smithi176:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T22:59:55.393 DEBUG:teuthology.orchestra.run.smithi176:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T22:59:55.394 DEBUG:teuthology.orchestra.run.smithi176:> zap_completion="0" 2022-03-08T22:59:55.394 DEBUG:teuthology.orchestra.run.smithi176:> while [ "$zap_completion" = "0" ] 2022-03-08T22:59:55.394 DEBUG:teuthology.orchestra.run.smithi176:> do 2022-03-08T22:59:55.395 DEBUG:teuthology.orchestra.run.smithi176:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T22:59:55.395 DEBUG:teuthology.orchestra.run.smithi176:> sleep 3s 2022-03-08T22:59:55.395 DEBUG:teuthology.orchestra.run.smithi176:> done 2022-03-08T22:59:55.396 DEBUG:teuthology.orchestra.run.smithi176:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T22:59:55.396 DEBUG:teuthology.orchestra.run.smithi176:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T22:59:55.396 DEBUG:teuthology.orchestra.run.smithi176:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T22:59:55.397 DEBUG:teuthology.orchestra.run.smithi176:> num_osd="0" 2022-03-08T22:59:55.397 DEBUG:teuthology.orchestra.run.smithi176:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T22:59:55.397 DEBUG:teuthology.orchestra.run.smithi176:> do 2022-03-08T22:59:55.398 DEBUG:teuthology.orchestra.run.smithi176:> echo "waiting for osd to come back up" 2022-03-08T22:59:55.398 DEBUG:teuthology.orchestra.run.smithi176:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T22:59:55.398 DEBUG:teuthology.orchestra.run.smithi176:> sleep 30s 2022-03-08T22:59:55.398 DEBUG:teuthology.orchestra.run.smithi176:> done 2022-03-08T22:59:55.399 DEBUG:teuthology.orchestra.run.smithi176:> ' 2022-03-08T22:59:55.413 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T22:59:55.414 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T22:59:55.415 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T22:59:56.074 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 1 16 1376 1360 84.9949 85 0.0114745 0.0116665 2022-03-08T22:59:57.074 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 2 16 2772 2756 86.1185 87.25 0.00552895 0.0115664 2022-03-08T22:59:57.936 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T22:59:58.074 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 3 16 4074 4058 84.5345 81.375 0.0164171 0.0117925 2022-03-08T22:59:58.758 INFO:teuthology.orchestra.run.smithi176.stderr:+ orig_num_osd=4 2022-03-08T22:59:58.758 INFO:teuthology.orchestra.run.smithi176.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T22:59:58.759 INFO:teuthology.orchestra.run.smithi176.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T22:59:58.854 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T22:59:59.075 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 4 16 5166 5150 80.4605 68.25 0.013197 0.0123974 2022-03-08T23:00:00.075 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 5 16 6441 6425 80.3044 79.6875 0.011442 0.0124268 2022-03-08T23:00:01.075 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 6 16 7531 7515 78.2735 68.125 0.013828 0.012757 2022-03-08T23:00:01.298 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:01.298433 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:00:01.298 INFO:tasks.rook.operator.smithi176.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T23:00:01.303 INFO:tasks.rook.operator.smithi176.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:00:01.303 INFO:tasks.rook.operator.smithi176.stdout:   ... // 3 identical fields 2022-03-08T23:00:01.303 INFO:tasks.rook.operator.smithi176.stdout:   Config: nil, 2022-03-08T23:00:01.304 INFO:tasks.rook.operator.smithi176.stdout:   Selection: {}, 2022-03-08T23:00:01.304 INFO:tasks.rook.operator.smithi176.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:00:01.304 INFO:tasks.rook.operator.smithi176.stdout:-  { 2022-03-08T23:00:01.304 INFO:tasks.rook.operator.smithi176.stdout:-  Name: "smithi176-lv-1", 2022-03-08T23:00:01.305 INFO:tasks.rook.operator.smithi176.stdout:-  Count: 1, 2022-03-08T23:00:01.305 INFO:tasks.rook.operator.smithi176.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:00:01.305 INFO:tasks.rook.operator.smithi176.stdout:-  }, 2022-03-08T23:00:01.305 INFO:tasks.rook.operator.smithi176.stdout:   {Name: "smithi176-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi176-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:01.306 INFO:tasks.rook.operator.smithi176.stdout:   {Name: "smithi176-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi176-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:01.306 INFO:tasks.rook.operator.smithi176.stdout:   {Name: "smithi176-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi176-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:01.306 INFO:tasks.rook.operator.smithi176.stdout:   }, 2022-03-08T23:00:01.307 INFO:tasks.rook.operator.smithi176.stdout:   }, 2022-03-08T23:00:01.307 INFO:tasks.rook.operator.smithi176.stdout:   Annotations: nil, 2022-03-08T23:00:01.307 INFO:tasks.rook.operator.smithi176.stdout:   Labels: nil, 2022-03-08T23:00:01.308 INFO:tasks.rook.operator.smithi176.stdout:   ... // 20 identical fields 2022-03-08T23:00:01.308 INFO:tasks.rook.operator.smithi176.stdout:  } 2022-03-08T23:00:01.308 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:01.298481 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:02.075 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 7 16 8485 8469 75.6071 59.625 0.0124151 0.0132038 2022-03-08T23:00:02.520 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:02.519908 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:00:03.075 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 8 16 9480 9464 73.9289 62.1875 0.0114867 0.0135 2022-03-08T23:00:04.076 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 9 16 10206 10190 70.7559 45.375 0.0274389 0.0141134 2022-03-08T23:00:04.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:04.845111 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:00:05.076 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 10 16 10428 10412 65.0675 13.875 0.00201636 0.0142582 2022-03-08T23:00:06.076 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 11 16 10428 10412 59.1523 0 - 0.0142582 2022-03-08T23:00:07.076 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 12 16 10428 10412 54.223 0 - 0.0142582 2022-03-08T23:00:08.076 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 13 16 10428 10412 50.0521 0 - 0.0142582 2022-03-08T23:00:09.076 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 14 16 10428 10412 46.4769 0 - 0.0142582 2022-03-08T23:00:10.077 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 15 16 10428 10412 43.3782 0 - 0.0142582 2022-03-08T23:00:10.394 INFO:teuthology.orchestra.run.smithi176.stdout:Success 2022-03-08T23:00:10.412 INFO:teuthology.orchestra.run.smithi176.stderr:+ removed_pv= 2022-03-08T23:00:10.412 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' '' = '' ']' 2022-03-08T23:00:10.412 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl get pv 2022-03-08T23:00:10.413 INFO:teuthology.orchestra.run.smithi176.stderr:++ grep Released 2022-03-08T23:00:10.413 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f1 -d ' ' 2022-03-08T23:00:10.460 INFO:teuthology.orchestra.run.smithi176.stderr:+ removed_pv= 2022-03-08T23:00:10.460 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 3s 2022-03-08T23:00:11.077 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 16 16 10428 10412 40.6671 0 - 0.0142582 2022-03-08T23:00:12.077 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 17 16 10626 10610 39.0027 1.76786 0.00207853 0.0237242 2022-03-08T23:00:12.354 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:12.354991 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T23:00:13.077 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 18 16 11051 11035 38.3108 26.5625 0.0191841 0.0260855 2022-03-08T23:00:13.461 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' '' = '' ']' 2022-03-08T23:00:13.462 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl get pv 2022-03-08T23:00:13.463 INFO:teuthology.orchestra.run.smithi176.stderr:++ grep Released 2022-03-08T23:00:13.463 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f1 -d ' ' 2022-03-08T23:00:13.579 INFO:teuthology.orchestra.run.smithi176.stderr:+ removed_pv=smithi176-lv-1 2022-03-08T23:00:13.580 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 3s 2022-03-08T23:00:14.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 19 16 11696 11680 38.4159 40.3125 0.0204801 0.0255669 2022-03-08T23:00:15.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:00:15.079267+0000 min lat: 0.00118928 max lat: 8.26063 avg lat: 0.0257206 2022-03-08T23:00:15.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:00:15.079 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 20 16 12452 12436 38.8573 47.25 0.00882891 0.0257206 2022-03-08T23:00:16.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 21 16 13278 13262 39.4651 51.625 0.0239363 0.0253258 2022-03-08T23:00:16.579 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' smithi176-lv-1 = '' ']' 2022-03-08T23:00:16.581 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl get pv smithi176-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-08T23:00:16.715 INFO:teuthology.orchestra.run.smithi176.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-08T23:00:16.716 INFO:teuthology.orchestra.run.smithi176.stderr:++ echo smithi176-lv-1 2022-03-08T23:00:16.716 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f1 -d - 2022-03-08T23:00:16.717 INFO:teuthology.orchestra.run.smithi176.stderr:+ host=smithi176 2022-03-08T23:00:16.718 INFO:teuthology.orchestra.run.smithi176.stderr:+ toolbox ceph orch device zap smithi176 /dev/vg_nvme/lv_1 --force 2022-03-08T23:00:16.718 INFO:teuthology.orchestra.run.smithi176.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi176 /dev/vg_nvme/lv_1 --force 2022-03-08T23:00:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:17.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 22 16 14162 14146 40.1823 55.25 0.0135217 0.0248716 2022-03-08T23:00:17.764 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_1 on smithi176 zapped 2022-03-08T23:00:17.835 INFO:teuthology.orchestra.run.smithi176.stderr:+ zap_completion=0 2022-03-08T23:00:17.835 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 0 = 0 ']' 2022-03-08T23:00:17.835 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T23:00:18.071 INFO:teuthology.orchestra.run.smithi176.stderr:+ zap_completion= 2022-03-08T23:00:18.072 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 3s 2022-03-08T23:00:18.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 23 16 14945 14929 40.5627 48.9375 0.0162344 0.0246383 2022-03-08T23:00:19.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 24 15 15696 15681 40.8307 47 0.0147181 0.0244775 2022-03-08T23:00:20.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 25 16 16196 16180 40.4449 31.1875 0.0111933 0.0242727 2022-03-08T23:00:21.072 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' '' = 0 ']' 2022-03-08T23:00:21.073 INFO:teuthology.orchestra.run.smithi176.stderr:+ kubectl patch pv smithi176-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-08T23:00:21.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 26 16 16933 16917 40.6607 46.0625 0.0172186 0.0245748 2022-03-08T23:00:21.617 INFO:teuthology.orchestra.run.smithi176.stdout:persistentvolume/smithi176-lv-1 patched 2022-03-08T23:00:21.620 INFO:teuthology.orchestra.run.smithi176.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:00:21.620 INFO:teuthology.orchestra.run.smithi176.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T23:00:22.078 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 27 16 17582 17566 40.6569 40.5625 0.0116457 0.0245803 2022-03-08T23:00:22.467 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:23.079 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 28 16 18293 18277 40.7918 44.4375 0.00945485 0.0244998 2022-03-08T23:00:24.079 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 29 16 19074 19058 41.0682 48.8125 0.0137369 0.0243364 2022-03-08T23:00:25.079 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 30 16 19690 19674 40.9825 38.5 0.0156913 0.0243896 2022-03-08T23:00:25.409 INFO:teuthology.orchestra.run.smithi176.stdout:['Success'] 2022-03-08T23:00:25.414 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:25.414010 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:00:25.419 INFO:tasks.rook.operator.smithi176.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-08T23:00:25.420 INFO:tasks.rook.operator.smithi176.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:00:25.420 INFO:tasks.rook.operator.smithi176.stdout:   ... // 3 identical fields 2022-03-08T23:00:25.420 INFO:tasks.rook.operator.smithi176.stdout:   Config: nil, 2022-03-08T23:00:25.421 INFO:tasks.rook.operator.smithi176.stdout:   Selection: {}, 2022-03-08T23:00:25.421 INFO:tasks.rook.operator.smithi176.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:00:25.421 INFO:tasks.rook.operator.smithi176.stdout:   {Name: "smithi176-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi176-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:25.422 INFO:tasks.rook.operator.smithi176.stdout:   {Name: "smithi176-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi176-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:25.422 INFO:tasks.rook.operator.smithi176.stdout:   {Name: "smithi176-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi176-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:00:25.422 INFO:tasks.rook.operator.smithi176.stdout:+  { 2022-03-08T23:00:25.422 INFO:tasks.rook.operator.smithi176.stdout:+  Name: "smithi176-lv-1", 2022-03-08T23:00:25.423 INFO:tasks.rook.operator.smithi176.stdout:+  Count: 1, 2022-03-08T23:00:25.423 INFO:tasks.rook.operator.smithi176.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:00:25.423 INFO:tasks.rook.operator.smithi176.stdout:+  }, 2022-03-08T23:00:25.423 INFO:tasks.rook.operator.smithi176.stdout:   }, 2022-03-08T23:00:25.424 INFO:tasks.rook.operator.smithi176.stdout:   }, 2022-03-08T23:00:25.424 INFO:tasks.rook.operator.smithi176.stdout:   Annotations: nil, 2022-03-08T23:00:25.424 INFO:tasks.rook.operator.smithi176.stdout:   Labels: nil, 2022-03-08T23:00:25.424 INFO:tasks.rook.operator.smithi176.stdout:   ... // 20 identical fields 2022-03-08T23:00:25.425 INFO:tasks.rook.operator.smithi176.stdout:  } 2022-03-08T23:00:25.445 INFO:teuthology.orchestra.run.smithi176.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:00:26.079 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 31 16 20362 20346 41.0152 42 0.0105467 0.0243561 2022-03-08T23:00:26.166 INFO:teuthology.orchestra.run.smithi176.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T23:00:26.865 INFO:teuthology.orchestra.run.smithi176.stdout:waiting for osd to come back up 2022-03-08T23:00:26.866 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=0 2022-03-08T23:00:26.866 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 0 '!=' 4 ']' 2022-03-08T23:00:26.867 INFO:teuthology.orchestra.run.smithi176.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:00:26.867 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T23:00:26.867 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:00:26.867 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T23:00:27.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 32 16 21137 21121 41.2463 48.4375 0.0235642 0.0242329 2022-03-08T23:00:27.301 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:00:28.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 33 16 21810 21794 41.2709 42.0625 0.0183548 0.0242187 2022-03-08T23:00:28.854 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:28.854589 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:00:28.855 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:28.854609 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:00:28.855 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:28.854617 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:00:28.900 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=3 2022-03-08T23:00:28.900 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 30s 2022-03-08T23:00:29.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 34 16 22443 22427 41.2205 39.5625 0.0306514 0.0242489 2022-03-08T23:00:29.463 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:29.459364 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:00:30.060 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:30.061036 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:00:30.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 35 16 23081 23065 41.182 39.875 0.0117712 0.02427 2022-03-08T23:00:30.176 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:30.176698 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:00:31.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 36 16 23494 23478 40.755 25.8125 0.0189236 0.0245248 2022-03-08T23:00:31.344 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:31.343629 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:00:31.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:31.462080 I | op-mon: start running mons 2022-03-08T23:00:31.483 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:31.482032 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:00:31.534 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:31.533133 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T23:00:31.622 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:31.622295 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:00:31.623 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:31.622498 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:00:32.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 37 15 24208 24193 40.8612 44.6875 0.0180781 0.0244633 2022-03-08T23:00:32.385 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:32.383767 I | op-mon: targeting the mon count 1 2022-03-08T23:00:32.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:32.407799 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-08T23:00:32.410 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:32.407899 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:00:33.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 38 16 24745 24729 40.6674 33.5 0.0125209 0.0245678 2022-03-08T23:00:33.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:33.242676 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:00:33.607 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:33.597182 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:00:34.080 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 39 16 25279 25263 40.4803 33.375 0.00936756 0.0246406 2022-03-08T23:00:34.345 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:34.345478 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T23:00:34.351 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:34.351596 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:00:35.081 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:00:35.081773+0000 min lat: 0.00118928 max lat: 8.26063 avg lat: 0.024992 2022-03-08T23:00:35.081 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:00:35.081 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 40 16 25610 25594 39.9855 20.6875 0.013503 0.024992 2022-03-08T23:00:36.081 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 41 16 26120 26104 39.7876 31.875 0.0339825 0.0251227 2022-03-08T23:00:36.484 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:36.482282 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:00:37.081 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 42 16 26647 26631 39.6244 32.9375 0.0318848 0.0252235 2022-03-08T23:00:38.082 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 43 16 27226 27210 39.5437 36.1875 0.00969728 0.0252777 2022-03-08T23:00:38.488 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:38.488008 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:00:39.082 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 44 16 27912 27896 39.6193 42.875 0.0146435 0.0252208 2022-03-08T23:00:40.084 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 45 16 28611 28595 39.7078 43.6875 0.0139828 0.0251713 2022-03-08T23:00:40.497 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:00:40.495020 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T23:00:41.084 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 46 16 29224 29208 39.6774 38.3125 0.0453735 0.0251898 2022-03-08T23:00:42.084 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 47 16 29804 29788 39.6044 36.25 0.0108301 0.0252377 2022-03-08T23:00:43.084 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 48 16 30376 30360 39.5241 35.75 0.0189136 0.0252918 2022-03-08T23:00:44.084 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 49 16 30923 30907 39.4151 34.1875 0.0154331 0.025361 2022-03-08T23:00:45.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 50 16 31596 31580 39.4672 42.0625 0.034565 0.0253205 2022-03-08T23:00:46.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 51 16 32203 32187 39.4372 37.9375 0.0273402 0.0253486 2022-03-08T23:00:47.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 52 16 32749 32733 39.3349 34.125 0.032193 0.0254117 2022-03-08T23:00:48.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 53 16 33347 33331 39.2979 37.375 0.0151634 0.0254348 2022-03-08T23:00:49.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 54 16 33892 33876 39.2009 34.0625 0.0566655 0.0254991 2022-03-08T23:00:50.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 55 16 34444 34428 39.1154 34.5 0.0283243 0.0255514 2022-03-08T23:00:51.086 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 56 16 34738 34722 38.745 18.375 0.0455769 0.0257661 2022-03-08T23:00:52.087 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 57 16 35313 35297 38.6957 35.9375 0.0276904 0.0258329 2022-03-08T23:00:53.087 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 58 16 35778 35762 38.5296 29.0625 0.0306394 0.0259424 2022-03-08T23:00:54.087 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 59 16 36314 36298 38.4443 33.5 0.137032 0.0259939 2022-03-08T23:00:55.090 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:00:55.090320+0000 min lat: 0.00118928 max lat: 8.26063 avg lat: 0.0259732 2022-03-08T23:00:55.090 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:00:55.091 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 60 16 36972 36956 38.487 41.125 0.0201131 0.0259732 2022-03-08T23:00:56.091 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 61 16 37600 37584 38.4986 39.25 0.0293006 0.0259654 2022-03-08T23:00:57.091 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 62 16 38199 38183 38.4815 37.4375 0.0182822 0.0259784 2022-03-08T23:00:58.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 63 16 38642 38626 38.3101 27.6875 0.0207059 0.0260918 2022-03-08T23:00:58.902 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:00:58.903 INFO:teuthology.orchestra.run.smithi176.stdout:waiting for osd to come back up 2022-03-08T23:00:58.904 INFO:teuthology.orchestra.run.smithi176.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:00:58.904 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T23:00:58.904 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:00:58.905 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T23:00:59.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 64 16 39019 39003 38.0797 23.5625 0.0234127 0.0262502 2022-03-08T23:00:59.443 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:00.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 65 16 39587 39571 38.0399 35.5 0.0255417 0.0262783 2022-03-08T23:01:00.274 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=3 2022-03-08T23:01:00.275 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 30s 2022-03-08T23:01:01.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 66 16 40076 40060 37.9266 30.5625 0.0134953 0.0263565 2022-03-08T23:01:02.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 67 16 40658 40642 37.9034 36.375 0.0257985 0.0263702 2022-03-08T23:01:03.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 68 16 41145 41129 37.7936 30.4375 0.00994018 0.0264484 2022-03-08T23:01:04.092 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 69 16 41676 41660 37.7268 33.1875 0.024096 0.0264644 2022-03-08T23:01:05.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 70 16 42210 42194 37.6646 33.375 0.0249657 0.0265398 2022-03-08T23:01:06.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 71 16 42801 42785 37.6543 36.9375 0.0165837 0.0265378 2022-03-08T23:01:07.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 72 16 43402 43386 37.653 37.5625 0.0203046 0.0265416 2022-03-08T23:01:08.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 73 15 43863 43848 37.5327 28.875 0.0465846 0.0266342 2022-03-08T23:01:09.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 74 16 44301 44285 37.3946 27.3125 0.0149339 0.0267241 2022-03-08T23:01:10.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 75 16 44743 44727 37.2643 27.625 0.0174199 0.0268237 2022-03-08T23:01:11.093 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 76 16 45207 45191 37.1555 29 0.00960213 0.0268675 2022-03-08T23:01:12.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 77 16 45249 45233 36.7071 2.625 0.00380614 0.0268484 2022-03-08T23:01:13.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 78 16 45249 45233 36.2365 0 - 0.0268484 2022-03-08T23:01:14.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 79 16 45249 45233 35.7779 0 - 0.0268484 2022-03-08T23:01:15.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:01:15.093693+0000 min lat: 0.00118928 max lat: 8.26063 avg lat: 0.0268484 2022-03-08T23:01:15.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:01:15.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 80 16 45249 45233 35.3307 0 - 0.0268484 2022-03-08T23:01:16.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 81 16 45249 45233 34.8946 0 - 0.0268484 2022-03-08T23:01:17.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 82 16 45249 45233 34.4691 0 - 0.0268484 2022-03-08T23:01:18.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 83 16 45249 45233 34.0539 0 - 0.0268484 2022-03-08T23:01:19.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 84 16 45249 45233 33.6485 0 - 0.0268484 2022-03-08T23:01:20.094 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 85 16 45249 45233 33.2527 0 - 0.0268484 2022-03-08T23:01:21.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 86 16 45249 45233 32.8661 0 - 0.0268484 2022-03-08T23:01:22.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 87 16 45249 45233 32.4883 0 - 0.0268484 2022-03-08T23:01:23.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 88 16 45249 45233 32.1192 0 - 0.0268484 2022-03-08T23:01:23.949 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:23.948455 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:01:23.950 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:23.948570 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:01:23.950 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:23.948634 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:01:24.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 89 16 45486 45470 31.9247 1.23438 0.014884 0.0312759 2022-03-08T23:01:24.838 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:24.835400 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:01:24.934 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:24.933392 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:24.938 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:24.933745 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:25.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 90 16 45983 45967 31.9151 31.0625 0.0616741 0.0313228 2022-03-08T23:01:25.722 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:25.720125 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:01:26.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 91 16 46450 46434 31.8851 29.1875 0.0375207 0.031352 2022-03-08T23:01:26.867 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:26.748667 I | op-mon: start running mons 2022-03-08T23:01:27.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 92 16 47004 46988 31.9149 34.625 0.0375244 0.0313227 2022-03-08T23:01:27.865 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:27.863533 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:01:28.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 93 16 47496 47480 31.9023 30.75 0.0119653 0.031336 2022-03-08T23:01:28.825 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:28.823044 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T23:01:29.095 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 94 16 48015 47999 31.908 32.4375 0.0233377 0.0313292 2022-03-08T23:01:29.314 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:29.311822 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:29.314 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:29.312015 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:29.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:29.330415 I | op-mon: targeting the mon count 1 2022-03-08T23:01:29.341 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:29.339864 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:30.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 95 16 48557 48541 31.9286 33.875 0.0316102 0.0312958 2022-03-08T23:01:30.276 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:01:30.276 INFO:teuthology.orchestra.run.smithi176.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:01:30.277 INFO:teuthology.orchestra.run.smithi176.stdout:waiting for osd to come back up 2022-03-08T23:01:30.278 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T23:01:30.278 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:01:30.278 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T23:01:30.390 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:01:30.705 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:30.701237 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:01:30.706 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:30.701263 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:31.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 96 16 48874 48858 31.8024 19.8125 0.0314911 0.0314277 2022-03-08T23:01:31.365 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=3 2022-03-08T23:01:31.365 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 30s 2022-03-08T23:01:31.447 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:31.446197 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:01:31.448 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:31.446226 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:32.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 97 16 49273 49257 31.7316 24.9375 0.021983 0.0315021 2022-03-08T23:01:32.127 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:32.125898 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:01:32.127 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:32.125926 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:33.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 98 16 49837 49821 31.7675 35.25 0.0176817 0.0314698 2022-03-08T23:01:33.482 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:33.480005 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:01:33.482 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:33.480024 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:01:34.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 99 16 50351 50335 31.7711 32.125 0.0359743 0.0314642 2022-03-08T23:01:34.128 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:34.122818 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:01:34.128 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:34.122839 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:01:34.768 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:34.766738 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:01:34.768 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:34.766755 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:01:34.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:34.766761 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-08T23:01:34.876 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:34.875024 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T23:01:35.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:01:35.095690+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0314603 2022-03-08T23:01:35.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:01:35.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 100 16 50874 50858 31.7802 32.6875 0.0119595 0.0314603 2022-03-08T23:01:35.204 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:35.196386 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:01:35.284 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:35.282527 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:01:35.313 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:35.311979 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:01:35.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:35.328229 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:01:35.338 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:35.332870 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:01:35.339 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:35.338349 I | op-mon: mons running: [a] 2022-03-08T23:01:36.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 101 16 51351 51335 31.7607 29.8125 0.0308942 0.0314747 2022-03-08T23:01:36.318 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:36.240933 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:01:36.319 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:36.240958 I | op-mon: mons created: 1 2022-03-08T23:01:37.060 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:37.058974 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:01:37.068 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:37.067110 I | op-mon: mons running: [a] 2022-03-08T23:01:37.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 102 16 51890 51874 31.7795 33.6875 0.0298873 0.0314558 2022-03-08T23:01:37.792 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:37.790757 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:01:37.792 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:37.790801 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:01:38.096 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 103 16 52376 52360 31.7659 30.375 0.032573 0.0314683 2022-03-08T23:01:38.666 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:38.663619 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:01:39.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 104 16 52829 52813 31.7326 28.3125 0.0383983 0.0315017 2022-03-08T23:01:39.515 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:39.508084 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:01:40.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 105 16 53283 53267 31.7006 28.375 0.0158661 0.031534 2022-03-08T23:01:40.358 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:40.350881 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:01:41.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 106 16 53691 53675 31.6421 25.5 0.0181221 0.0315931 2022-03-08T23:01:41.305 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:41.304101 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:01:41.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:41.304118 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:01:42.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 107 16 54200 54184 31.6437 31.8125 0.0336328 0.0315929 2022-03-08T23:01:42.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:42.107571 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:01:42.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:42.850607 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:01:42.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:42.850633 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:01:42.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:42.850639 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:01:43.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 108 16 54612 54596 31.5891 25.75 0.0364363 0.0316472 2022-03-08T23:01:43.587 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:43.584637 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:01:43.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:43.618125 I | op-mgr: start running mgr 2022-03-08T23:01:43.621 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:43.618147 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:01:44.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 109 16 55138 55122 31.6009 32.875 0.0298681 0.0316342 2022-03-08T23:01:44.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.411413 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:01:44.422 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.418979 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:01:44.429 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.423753 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:01:44.543 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.541958 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:01:44.543 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.541983 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:01:44.571 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.569059 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:01:44.575 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.573692 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-2" 2022-03-08T23:01:44.583 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.580182 I | op-osd: OSD PVC "smithi176-lv-2-data-0spfs6" already exists 2022-03-08T23:01:44.583 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.580212 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-3" 2022-03-08T23:01:44.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.580223 I | op-osd: OSD PVC "smithi176-lv-3-data-0w78wp" already exists 2022-03-08T23:01:44.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.580231 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-4" 2022-03-08T23:01:44.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.580240 I | op-osd: OSD PVC "smithi176-lv-4-data-0dsmg4" already exists 2022-03-08T23:01:44.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.580249 I | op-osd: creating 1 new PVCs for device set "smithi176-lv-1" 2022-03-08T23:01:44.593 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.591756 I | op-osd: successfully provisioned PVC "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:01:44.624 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.622402 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-2-data-0spfs6" 2022-03-08T23:01:44.624 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.622466 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-3-data-0w78wp" 2022-03-08T23:01:44.624 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.622476 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-4-data-0dsmg4" 2022-03-08T23:01:44.625 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.622482 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:01:44.859 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.857799 I | op-osd: started OSD provisioning job for PVC "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:01:44.860 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.857868 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:01:44.860 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.857896 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:01:44.938 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:44.936905 I | op-osd: OSD orchestration status for PVC smithi176-lv-1-data-0gt5tk is "starting" 2022-03-08T23:01:45.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 110 16 55553 55537 31.5494 25.9375 0.0371564 0.0316848 2022-03-08T23:01:46.097 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 111 16 55932 55916 31.4785 23.6875 0.0311538 0.0317593 2022-03-08T23:01:47.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 112 16 56393 56377 31.4547 28.8125 0.0327423 0.0317828 2022-03-08T23:01:47.169 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:47.164208 I | op-mgr: successful modules: dashboard 2022-03-08T23:01:47.169 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:47.164241 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:01:47.173 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:47.164247 I | op-mgr: successful modules: prometheus 2022-03-08T23:01:48.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 113 16 56764 56748 31.3815 23.1875 0.0244362 0.0318493 2022-03-08T23:01:48.627 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:48.624025 I | op-mgr: successful modules: balancer 2022-03-08T23:01:48.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:48.647636 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:01:48.661 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:48.657358 I | op-osd: updating OSD 1 on PVC "smithi176-lv-3-data-0w78wp" 2022-03-08T23:01:48.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:48.657385 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-3-data-0w78wp" 2022-03-08T23:01:49.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 114 16 57263 57247 31.3798 31.1875 0.0234581 0.0318569 2022-03-08T23:01:50.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 115 16 57718 57702 31.3542 28.4375 0.036529 0.0318816 2022-03-08T23:01:51.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 116 16 58134 58118 31.308 26 0.0239158 0.0319303 2022-03-08T23:01:51.621 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:51.619866 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:01:52.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 117 16 58517 58501 31.245 23.9375 0.0509299 0.031985 2022-03-08T23:01:52.993 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:52.988599 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:01:52.997 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:52.994349 I | op-osd: updating OSD 2 on PVC "smithi176-lv-4-data-0dsmg4" 2022-03-08T23:01:52.997 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:52.994375 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-4-data-0dsmg4" 2022-03-08T23:01:53.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 118 16 59001 58985 31.2365 30.25 0.0611529 0.0320046 2022-03-08T23:01:54.098 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 119 16 59509 59493 31.2408 31.75 0.0239981 0.0319989 2022-03-08T23:01:55.099 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:01:55.098517+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.031994 2022-03-08T23:01:55.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:01:55.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 120 16 60022 60006 31.2474 32.0625 0.0279904 0.031994 2022-03-08T23:01:56.099 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 121 16 60483 60467 31.2273 28.8125 0.0244858 0.032007 2022-03-08T23:01:56.360 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:56.358550 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:01:56.408 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:56.405931 I | op-osd: updating OSD 3 on PVC "smithi176-lv-2-data-0spfs6" 2022-03-08T23:01:56.408 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:01:56.406001 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-2-data-0spfs6" 2022-03-08T23:01:57.099 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 122 16 60999 60983 31.2356 32.25 0.0450564 0.0320052 2022-03-08T23:01:58.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 123 16 61522 61506 31.2474 32.6875 0.0401079 0.0319933 2022-03-08T23:01:59.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 124 16 62026 62010 31.2494 31.5 0.034166 0.0319915 2022-03-08T23:02:00.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 125 16 62564 62548 31.2684 33.625 0.0246727 0.0319724 2022-03-08T23:02:01.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 126 16 63008 62992 31.2405 27.75 0.0361107 0.032 2022-03-08T23:02:01.366 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:02:01.367 INFO:teuthology.orchestra.run.smithi176.stdout:waiting for osd to come back up 2022-03-08T23:02:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:02:01.368 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T23:02:01.368 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:02:01.368 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T23:02:01.828 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:02:02.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 127 15 63456 63441 31.2154 28.0625 0.0562233 0.0320256 2022-03-08T23:02:02.697 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=3 2022-03-08T23:02:02.697 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 30s 2022-03-08T23:02:03.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 128 16 63914 63898 31.1947 28.5625 0.0668995 0.0320479 2022-03-08T23:02:04.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 129 16 64340 64324 31.1592 26.625 0.0240764 0.0320839 2022-03-08T23:02:05.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 130 16 64877 64861 31.1777 33.5625 0.0186564 0.0320645 2022-03-08T23:02:06.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 131 16 65349 65333 31.1649 29.5 0.0288867 0.032059 2022-03-08T23:02:07.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 132 16 65723 65707 31.1058 23.375 0.0229642 0.0321386 2022-03-08T23:02:08.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 133 16 66191 66175 31.0919 29.25 0.0260577 0.0321553 2022-03-08T23:02:09.051 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:09.048221 I | op-osd: OSD orchestration status for node smithi176-lv-1-data-0gt5tk is "orchestrating" 2022-03-08T23:02:09.100 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 134 15 66698 66683 31.0967 31.75 0.015901 0.0321503 2022-03-08T23:02:09.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:09.583969 I | op-osd: OSD orchestration status for PVC smithi176-lv-1-data-0gt5tk is "orchestrating" 2022-03-08T23:02:10.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 135 16 67131 67115 31.0664 27 0.00901019 0.0321374 2022-03-08T23:02:11.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 136 16 67173 67157 30.8573 2.625 0.00891606 0.0321214 2022-03-08T23:02:12.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 137 16 67173 67157 30.632 0 - 0.0321214 2022-03-08T23:02:13.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 138 16 67173 67157 30.4101 0 - 0.0321214 2022-03-08T23:02:14.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 139 16 67173 67157 30.1913 0 - 0.0321214 2022-03-08T23:02:15.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:02:15.100509+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0321214 2022-03-08T23:02:15.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:02:15.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 140 16 67173 67157 29.9757 0 - 0.0321214 2022-03-08T23:02:15.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:15.640000 I | op-osd: OSD orchestration status for PVC smithi176-lv-1-data-0gt5tk is "completed" 2022-03-08T23:02:15.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:15.640038 I | op-osd: creating OSD 0 on PVC "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:02:15.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:15.640047 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:02:16.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 141 16 67173 67157 29.7631 0 - 0.0321214 2022-03-08T23:02:16.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:16.785079 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:02:16.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:16.785097 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:17.013 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:17.011604 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:02:17.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 142 16 67173 67157 29.5535 0 - 0.0321214 2022-03-08T23:02:18.082 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:18.080622 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:02:18.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 143 16 67173 67157 29.3469 0 - 0.0321214 2022-03-08T23:02:18.255 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:18.253471 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-08T23:02:19.101 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 144 16 67173 67157 29.1431 0 - 0.0321214 2022-03-08T23:02:20.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 145 16 67445 67429 29.0593 1.88889 0.0216583 0.0344023 2022-03-08T23:02:21.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 146 16 67905 67889 29.0572 28.75 0.0360472 0.0344062 2022-03-08T23:02:22.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 147 16 68435 68419 29.0848 33.125 0.0320516 0.0343721 2022-03-08T23:02:23.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 148 16 68854 68838 29.0653 26.1875 0.0642849 0.0343941 2022-03-08T23:02:24.102 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 149 16 69298 69282 29.0564 27.75 0.0700358 0.0344043 2022-03-08T23:02:25.103 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 150 16 69771 69755 29.0596 29.5625 0.0165161 0.0344025 2022-03-08T23:02:26.103 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 151 16 70159 70143 29.0277 24.25 0.0406041 0.0344411 2022-03-08T23:02:27.103 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 152 16 70531 70515 28.9897 23.25 0.0152024 0.0344554 2022-03-08T23:02:28.103 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 153 16 71013 70997 28.9971 30.125 0.015263 0.0344768 2022-03-08T23:02:29.104 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 154 16 71481 71465 28.9986 29.25 0.0304357 0.0344742 2022-03-08T23:02:30.104 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 155 16 71947 71931 28.9994 29.125 0.0337119 0.0344737 2022-03-08T23:02:31.104 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 156 16 72346 72330 28.9733 24.9375 0.107887 0.0345046 2022-03-08T23:02:32.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 157 16 72843 72827 28.9866 31.0625 0.0194001 0.0344889 2022-03-08T23:02:32.698 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:02:32.698 INFO:teuthology.orchestra.run.smithi176.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:02:32.699 INFO:teuthology.orchestra.run.smithi176.stdout:waiting for osd to come back up 2022-03-08T23:02:32.699 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T23:02:32.700 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:02:32.700 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T23:02:32.800 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:02:33.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 158 16 73168 73152 28.9317 20.3125 0.0745074 0.0345501 2022-03-08T23:02:33.659 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=3 2022-03-08T23:02:33.659 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 30s 2022-03-08T23:02:34.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 159 16 73522 73506 28.8889 22.125 0.0214895 0.0346 2022-03-08T23:02:35.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:02:35.104066+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0346247 2022-03-08T23:02:35.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:02:35.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 160 16 73943 73927 28.8728 26.3125 0.0430479 0.0346247 2022-03-08T23:02:36.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 161 16 74360 74344 28.8553 26.0625 0.0282205 0.0346464 2022-03-08T23:02:37.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 162 16 74827 74811 28.8573 29.1875 0.0248657 0.0346445 2022-03-08T23:02:38.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 163 16 75223 75207 28.8321 24.75 0.0640651 0.0346552 2022-03-08T23:02:39.105 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 164 16 75635 75619 28.8133 25.75 0.0416222 0.0346941 2022-03-08T23:02:40.106 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 165 16 76042 76026 28.7927 25.4375 0.0209497 0.0347054 2022-03-08T23:02:40.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:40.503807 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11002 quincy" 2022-03-08T23:02:40.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:40.503828 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:02:40.506 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:40.503837 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-08T23:02:41.106 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 166 16 76471 76455 28.7808 26.8125 0.0269219 0.0347326 2022-03-08T23:02:41.292 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:41.290840 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:02:41.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:41.338197 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:41.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:41.338510 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:42.106 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 167 16 76790 76774 28.7278 19.9375 0.0406951 0.0347993 2022-03-08T23:02:42.336 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:42.334466 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-08T23:02:42.695 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:42.693896 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:02:42.630524476 +0000 UTC m=+528.619889602 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:02:42.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:42.704277 I | op-mon: start running mons 2022-03-08T23:02:42.895 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:42.893645 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:02:43.106 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 168 16 77151 77135 28.6911 22.5625 0.00853714 0.0348376 2022-03-08T23:02:43.540 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:43.537806 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T23:02:44.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 169 16 77530 77514 28.6614 23.6875 0.0487169 0.0348811 2022-03-08T23:02:44.169 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:44.166305 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:44.303 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:44.295399 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:44.646 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:44.644634 I | op-mon: targeting the mon count 1 2022-03-08T23:02:44.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:44.648589 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:02:45.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 170 16 77949 77933 28.6469 26.1875 0.0495453 0.0348982 2022-03-08T23:02:45.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:45.215580 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:02:45.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:45.215598 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:02:45.988 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:45.985816 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:02:45.988 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:45.985843 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:02:46.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 171 16 78353 78337 28.627 25.25 0.0339855 0.0349224 2022-03-08T23:02:46.803 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:46.801788 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:02:46.803 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:46.801814 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:02:47.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 172 16 78679 78663 28.579 20.375 0.0602326 0.0349746 2022-03-08T23:02:47.564 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:47.561878 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:02:47.564 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:47.561902 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:02:48.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 173 16 79090 79074 28.5623 25.6875 0.0264163 0.0350005 2022-03-08T23:02:48.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:48.328321 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-08T23:02:48.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:48.328339 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:02:49.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 174 16 79512 79496 28.5497 26.375 0.0232821 0.035017 2022-03-08T23:02:49.496 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:49.310399 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:02:49.496 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:49.310423 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:02:49.497 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:49.310429 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-08T23:02:49.772 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:49.771065 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.176:6789"]}] data:a=172.21.15.176:6789 mapping:{"node":{"a":{"Name":"smithi176","Hostname":"smithi176","Address":"172.21.15.176"}}} maxMonId:0] 2022-03-08T23:02:50.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 175 16 79873 79857 28.5155 22.5625 0.0433091 0.0350594 2022-03-08T23:02:50.143 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:50.142252 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:02:50.155 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:50.152390 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:02:50.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:50.659244 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:02:50.693 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:50.689450 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:02:50.693 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:50.689481 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:50.890 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:50.888484 I | op-mon: mons running: [a] 2022-03-08T23:02:51.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 176 16 80265 80249 28.4927 24.5 0.146094 0.0350868 2022-03-08T23:02:51.559 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:51.490924 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:51.562 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:51.490987 I | op-mon: mons created: 1 2022-03-08T23:02:52.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 177 16 80608 80592 28.4528 21.4375 0.0371105 0.0351375 2022-03-08T23:02:52.179 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:52.172147 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:02:52.181 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:52.178991 I | op-mon: mons running: [a] 2022-03-08T23:02:52.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:52.902905 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:02:52.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:52.902941 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:02:53.107 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 178 16 80984 80968 28.425 23.5 0.0294967 0.0351706 2022-03-08T23:02:53.648 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:53.646943 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:02:54.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 179 16 81388 81372 28.4072 25.25 0.0394875 0.035192 2022-03-08T23:02:54.524 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:54.522232 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:02:55.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:02:55.106974+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0352152 2022-03-08T23:02:55.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:02:55.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 180 16 81792 81776 28.3897 25.25 0.0746876 0.0352152 2022-03-08T23:02:55.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:55.232819 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:02:56.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 181 16 82209 82193 28.3768 26.0625 0.0205998 0.0352311 2022-03-08T23:02:56.186 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:56.185063 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:02:56.187 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:56.185085 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:02:56.795 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:56.794112 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:02:57.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 182 16 82379 82363 28.2792 10.625 0.0304476 0.0352403 2022-03-08T23:02:57.196 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:57.194394 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:02:57.196 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:57.194429 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:02:57.196 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:57.194437 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:02:57.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:57.650819 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:02:57.681 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:57.680609 I | op-mgr: start running mgr 2022-03-08T23:02:57.682 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:57.680638 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:02:58.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 183 16 82379 82363 28.1247 0 - 0.0352403 2022-03-08T23:02:58.142 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.140236 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:02:58.151 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.149795 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:02:58.205 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.203849 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:02:58.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.290300 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:02:58.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.290336 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:02:58.352 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.351108 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:02:58.356 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354682 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-2" 2022-03-08T23:02:58.356 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354723 I | op-osd: OSD PVC "smithi176-lv-2-data-0spfs6" already exists 2022-03-08T23:02:58.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354736 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-3" 2022-03-08T23:02:58.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354747 I | op-osd: OSD PVC "smithi176-lv-3-data-0w78wp" already exists 2022-03-08T23:02:58.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354754 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-4" 2022-03-08T23:02:58.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354763 I | op-osd: OSD PVC "smithi176-lv-4-data-0dsmg4" already exists 2022-03-08T23:02:58.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354770 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi176-lv-1" 2022-03-08T23:02:58.358 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.354798 I | op-osd: OSD PVC "smithi176-lv-1-data-0gt5tk" already exists 2022-03-08T23:02:58.367 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.366159 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-2-data-0spfs6" 2022-03-08T23:02:58.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.366181 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-3-data-0w78wp" 2022-03-08T23:02:58.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.366209 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-4-data-0dsmg4" 2022-03-08T23:02:58.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.366366 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:02:58.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.366392 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:02:58.369 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:58.366412 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:02:59.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 184 16 82379 82363 27.9719 0 - 0.0352403 2022-03-08T23:02:59.580 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:59.578770 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:02:59.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:59.583098 I | op-osd: updating OSD 0 on PVC "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:02:59.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:59.583116 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-1-data-0gt5tk" 2022-03-08T23:02:59.888 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:59.886049 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:02:59.897 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:59.890038 I | op-mgr: successful modules: dashboard 2022-03-08T23:02:59.897 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:02:59.890717 I | op-mgr: successful modules: prometheus 2022-03-08T23:03:00.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 185 16 82379 82363 27.8207 0 - 0.0352403 2022-03-08T23:03:00.495 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:00.490238 I | op-mgr: successful modules: balancer 2022-03-08T23:03:01.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 186 16 82382 82366 27.6721 0.046875 0.0324902 0.0352946 2022-03-08T23:03:01.818 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:01.816430 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:01.825 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:01.823601 I | op-osd: updating OSD 1 on PVC "smithi176-lv-3-data-0w78wp" 2022-03-08T23:03:01.826 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:01.823667 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-3-data-0w78wp" 2022-03-08T23:03:01.907 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:01.905603 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:03:02.108 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 187 16 82489 82473 27.5599 6.6875 0.0204994 0.0362671 2022-03-08T23:03:03.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 188 16 82762 82746 27.5041 17.0625 0.0412007 0.0363459 2022-03-08T23:03:03.660 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 3 '!=' 4 ']' 2022-03-08T23:03:03.661 INFO:teuthology.orchestra.run.smithi176.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:03:03.661 INFO:teuthology.orchestra.run.smithi176.stdout:waiting for osd to come back up 2022-03-08T23:03:03.662 INFO:teuthology.orchestra.run.smithi176.stderr:++ toolbox ceph osd stat 2022-03-08T23:03:03.662 INFO:teuthology.orchestra.run.smithi176.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:03:03.662 INFO:teuthology.orchestra.run.smithi176.stderr:++ cut -f3 -d ' ' 2022-03-08T23:03:03.774 INFO:teuthology.orchestra.run.smithi176.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:03:03.997 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:03.995868 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:04.006 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:04.004431 I | op-osd: updating OSD 2 on PVC "smithi176-lv-4-data-0dsmg4" 2022-03-08T23:03:04.008 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:04.006689 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-4-data-0dsmg4" 2022-03-08T23:03:04.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 189 16 83036 83020 27.4491 17.125 0.0791633 0.0364211 2022-03-08T23:03:04.669 INFO:teuthology.orchestra.run.smithi176.stderr:+ num_osd=4 2022-03-08T23:03:04.670 INFO:teuthology.orchestra.run.smithi176.stderr:+ sleep 30s 2022-03-08T23:03:05.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 190 16 83278 83262 27.3843 15.125 0.070472 0.0365054 2022-03-08T23:03:06.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 191 16 83564 83548 27.3345 17.875 0.0461981 0.0365696 2022-03-08T23:03:06.369 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:06.352199 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-08T23:03:06.369 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:06.361296 I | op-osd: updating OSD 3 on PVC "smithi176-lv-2-data-0spfs6" 2022-03-08T23:03:06.369 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:06.361328 I | op-osd: OSD will have its main bluestore block on "smithi176-lv-2-data-0spfs6" 2022-03-08T23:03:07.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 192 16 83836 83820 27.2807 17 0.0587751 0.036644 2022-03-08T23:03:07.269 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:07.267950 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:03:07.270 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:07.267973 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:03:08.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 193 16 84122 84106 27.2319 17.875 0.0379705 0.0367078 2022-03-08T23:03:09.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 194 16 84405 84389 27.1827 17.6875 0.0569977 0.0367792 2022-03-08T23:03:10.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 195 16 84660 84644 27.125 15.9375 0.0338115 0.0368343 2022-03-08T23:03:11.109 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 196 16 84870 84854 27.0536 13.125 0.0359056 0.0369522 2022-03-08T23:03:12.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 197 16 85171 85155 27.0118 18.8125 0.0362708 0.037011 2022-03-08T23:03:13.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 198 16 85467 85451 26.9688 18.5 0.0421932 0.0370692 2022-03-08T23:03:14.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 199 16 85780 85764 26.9315 19.5625 0.0162318 0.0371191 2022-03-08T23:03:15.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:03:15.109145+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0371839 2022-03-08T23:03:15.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:03:15.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 200 16 86064 86048 26.8856 17.75 0.0526024 0.0371839 2022-03-08T23:03:16.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 201 16 86383 86367 26.851 19.9375 0.0195777 0.0372296 2022-03-08T23:03:17.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 202 16 86677 86661 26.8091 18.375 0.0399172 0.0372893 2022-03-08T23:03:18.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 203 16 86960 86944 26.7641 17.6875 0.102866 0.0373513 2022-03-08T23:03:19.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 204 16 87189 87173 26.7031 14.3125 0.0127651 0.0373984 2022-03-08T23:03:20.110 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 205 16 87397 87381 26.6363 13 0.0311097 0.0374945 2022-03-08T23:03:21.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 206 16 87597 87581 26.5676 12.5 0.0379914 0.0376279 2022-03-08T23:03:22.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 207 16 87891 87875 26.528 18.375 0.0285826 0.0376816 2022-03-08T23:03:23.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 208 16 88185 88169 26.4888 18.375 0.0361389 0.0377422 2022-03-08T23:03:24.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 209 16 88457 88441 26.4434 17 0.0561826 0.0378078 2022-03-08T23:03:25.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 210 16 88701 88685 26.3901 15.25 0.0585321 0.0378828 2022-03-08T23:03:26.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 211 16 88925 88909 26.3314 14 0.0160188 0.0379621 2022-03-08T23:03:27.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 212 16 89197 89181 26.2874 17 0.0954583 0.0380298 2022-03-08T23:03:28.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 213 16 89442 89426 26.2359 15.3125 0.0457754 0.0381056 2022-03-08T23:03:29.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 214 16 89742 89726 26.2009 18.75 0.0282047 0.0381557 2022-03-08T23:03:30.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 215 16 90052 90036 26.1691 19.375 0.0172502 0.0382014 2022-03-08T23:03:31.111 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 216 16 90176 90160 26.0838 7.75 0.0685107 0.0383263 2022-03-08T23:03:32.112 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 217 16 90470 90454 26.0483 18.375 0.0231742 0.0383794 2022-03-08T23:03:33.112 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 218 16 90754 90738 26.0102 17.75 0.095069 0.0384336 2022-03-08T23:03:34.113 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 219 16 91063 91047 25.9795 19.3125 0.0436801 0.0384812 2022-03-08T23:03:34.670 INFO:teuthology.orchestra.run.smithi176.stderr:+ '[' 4 '!=' 4 ']' 2022-03-08T23:03:34.672 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:03:34.687 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch status 2022-03-08T23:03:35.113 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:03:35.112408+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0385355 2022-03-08T23:03:35.114 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:03:35.114 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 220 16 91336 91320 25.939 17.0625 0.145184 0.0385355 2022-03-08T23:03:35.785 INFO:teuthology.orchestra.run.smithi176.stdout:Backend: rook 2022-03-08T23:03:35.785 INFO:teuthology.orchestra.run.smithi176.stdout:Available: Yes 2022-03-08T23:03:35.819 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ps 2022-03-08T23:03:36.113 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 221 16 91590 91574 25.8934 15.875 0.0310253 0.0386084 2022-03-08T23:03:36.833 INFO:teuthology.orchestra.run.smithi176.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:03:36.834 INFO:teuthology.orchestra.run.smithi176.stdout:crashcollector.crash smithi176 running (6m) 0s ago 6m - - 1c5288aef625 2022-03-08T23:03:36.834 INFO:teuthology.orchestra.run.smithi176.stdout:mgr.a smithi176 running (8m) 0s ago 8m - - 1c5288aef625 2022-03-08T23:03:36.834 INFO:teuthology.orchestra.run.smithi176.stdout:mon.a smithi176 running (8m) 0s ago 8m - - 1c5288aef625 2022-03-08T23:03:36.834 INFO:teuthology.orchestra.run.smithi176.stdout:osd.0 smithi176 running (80s) 0s ago 80s - - 1c5288aef625 2022-03-08T23:03:36.835 INFO:teuthology.orchestra.run.smithi176.stdout:osd.1 smithi176 running (6m) 0s ago 6m - - 1c5288aef625 2022-03-08T23:03:36.835 INFO:teuthology.orchestra.run.smithi176.stdout:osd.2 smithi176 running (6m) 0s ago 6m - - 1c5288aef625 2022-03-08T23:03:36.835 INFO:teuthology.orchestra.run.smithi176.stdout:osd.3 smithi176 running (6m) 0s ago 6m - - 1c5288aef625 2022-03-08T23:03:36.870 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls 2022-03-08T23:03:37.113 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 222 16 91864 91848 25.8539 17.125 0.078168 0.038667 2022-03-08T23:03:38.041 INFO:teuthology.orchestra.run.smithi176.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:03:38.041 INFO:teuthology.orchestra.run.smithi176.stdout:crash 1/1 0s ago 6m * 2022-03-08T23:03:38.042 INFO:teuthology.orchestra.run.smithi176.stdout:mgr 1/1 0s ago 8m count:1 2022-03-08T23:03:38.043 INFO:teuthology.orchestra.run.smithi176.stdout:mon 1/1 0s ago 8m count:1 2022-03-08T23:03:38.043 INFO:teuthology.orchestra.run.smithi176.stdout:osd 8 0s ago 6m 2022-03-08T23:03:38.043 INFO:teuthology.orchestra.run.smithi176.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:03:38.089 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch host ls 2022-03-08T23:03:38.113 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 223 16 92108 92092 25.8064 15.25 0.0587007 0.0387377 2022-03-08T23:03:39.005 INFO:teuthology.orchestra.run.smithi176.stdout:HOST ADDR LABELS STATUS 2022-03-08T23:03:39.006 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 172.21.15.176/smithi176 2022-03-08T23:03:39.006 INFO:teuthology.orchestra.run.smithi176.stdout:1 hosts in cluster 2022-03-08T23:03:39.033 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch device ls 2022-03-08T23:03:39.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 224 16 92330 92314 25.7531 13.875 0.0698408 0.0388183 2022-03-08T23:03:39.977 INFO:teuthology.orchestra.run.smithi176.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:03:39.978 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:03:39.978 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:03:39.978 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:03:39.979 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:03:40.013 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch apply rgw foo 2022-03-08T23:03:40.114 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 225 16 92567 92551 25.7045 14.8125 0.0482758 0.0388893 2022-03-08T23:03:41.114 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 226 16 92811 92795 25.6582 15.25 0.0651903 0.0389617 2022-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi176.stdout:Created 2022-03-08T23:03:41.256 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:41.254649 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T23:03:41.290 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch apply mds foo 2022-03-08T23:03:41.938 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:41.936919 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:03:42.114 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 227 16 92922 92906 25.5757 6.9375 0.0397394 0.0390742 2022-03-08T23:03:42.825 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:42.823096 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:03:42.829 INFO:teuthology.orchestra.run.smithi176.stdout:Created 2022-03-08T23:03:42.893 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch apply rbd-mirror 2022-03-08T23:03:42.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:42.928964 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:03:43.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 228 16 93085 93069 25.5081 10.1875 0.113218 0.0391776 2022-03-08T23:03:43.853 INFO:teuthology.orchestra.run.smithi176.stdout:Success 2022-03-08T23:03:43.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:43.873467 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:03:43.892 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch apply nfs foo --port 12777 2022-03-08T23:03:44.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 229 16 93238 93222 25.4384 9.5625 0.0518355 0.0392827 2022-03-08T23:03:44.227 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:44.222709 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:03:44.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:44.344003 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:03:44.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:44.344030 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:03:45.065 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:45.063003 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T23:03:45.066 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:45.063033 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T23:03:45.066 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:45.063046 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T23:03:45.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 230 16 93424 93408 25.3784 11.625 0.0588253 0.0393898 2022-03-08T23:03:45.785 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:45.781476 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:03:46.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 231 16 93610 93594 25.3188 11.625 0.0334759 0.0394672 2022-03-08T23:03:46.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:46.899080 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T23:03:47.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 232 16 93727 93711 25.2412 7.3125 0.0675764 0.0395999 2022-03-08T23:03:47.286 INFO:teuthology.orchestra.run.smithi176.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T23:03:47.287 INFO:teuthology.orchestra.run.smithi176.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T23:03:47.363 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-08T23:03:47.378 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-08T23:03:47.378 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-08T23:03:47.416 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:47.413920 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi176": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:47.576 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:47.572011 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi176": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:03:48.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 233 16 93819 93803 25.1576 5.75 0.519612 0.0397192 2022-03-08T23:03:49.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 234 16 94039 94023 25.1088 13.75 0.0524563 0.0398131 2022-03-08T23:03:50.090 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:03:50.089100 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T23:03:50.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 235 16 94265 94249 25.0621 14.125 0.0536266 0.0398867 2022-03-08T23:03:51.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 236 16 94489 94473 25.0152 14 0.0214582 0.0399609 2022-03-08T23:03:52.116 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 237 16 94732 94716 24.9737 15.1875 0.0279314 0.0400312 2022-03-08T23:03:53.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 238 16 94927 94911 24.92 12.1875 0.114894 0.0401091 2022-03-08T23:03:54.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 239 16 95124 95108 24.8672 12.3125 0.0530747 0.0402023 2022-03-08T23:03:55.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:03:55.116021+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0402708 2022-03-08T23:03:55.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:03:55.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 240 16 95353 95337 24.8233 14.3125 0.0242628 0.0402708 2022-03-08T23:03:56.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 241 16 95555 95539 24.7727 12.625 0.115361 0.0403415 2022-03-08T23:03:57.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 242 16 95749 95733 24.7204 12.125 0.0660522 0.0404386 2022-03-08T23:03:58.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 243 16 95962 95946 24.6734 13.3125 0.191018 0.0405098 2022-03-08T23:03:59.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 244 16 96243 96227 24.6443 17.5625 0.0903288 0.0405681 2022-03-08T23:04:00.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 245 16 96442 96426 24.5945 12.4375 0.0541865 0.0406442 2022-03-08T23:04:01.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 246 15 96627 96612 24.5417 11.625 0.278168 0.0407249 2022-03-08T23:04:02.117 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 247 16 96869 96853 24.5034 15.0625 0.0620053 0.0408016 2022-03-08T23:04:03.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 248 16 97118 97102 24.4673 15.5625 0.0753531 0.0408608 2022-03-08T23:04:03.625 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:03.621591 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-08T23:04:03.631 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:03.625864 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-08T23:04:03.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:03.633001 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-08T23:04:03.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:03.651395 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-08T23:04:03.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:03.653645 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-08T23:04:03.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:03.656792 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-08T23:04:04.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 249 16 97293 97277 24.413 10.9375 0.0741922 0.0409477 2022-03-08T23:04:05.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 250 16 97487 97471 24.3638 12.125 0.0829333 0.0410332 2022-03-08T23:04:06.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 251 16 97774 97758 24.3382 17.9375 0.0418974 0.0410766 2022-03-08T23:04:06.709 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:06.699683 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-08T23:04:07.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 252 16 97977 97961 24.292 12.6875 0.0534122 0.0411502 2022-03-08T23:04:08.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 253 16 98130 98114 24.2337 9.5625 0.0714784 0.041249 2022-03-08T23:04:09.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 254 16 98388 98372 24.2018 16.125 0.0948328 0.0412952 2022-03-08T23:04:09.729 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.727681 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-08T23:04:09.757 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.752225 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-08T23:04:09.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.768219 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-08T23:04:09.806 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.803698 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-08T23:04:09.854 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.847591 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T23:04:09.953 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.951621 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:09.954 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.951647 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:09.993 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:09.986838 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:10.118 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 255 16 98630 98614 24.1662 15.125 0.0467943 0.0413658 2022-03-08T23:04:11.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 256 16 98840 98824 24.1231 13.125 0.119076 0.0414398 2022-03-08T23:04:12.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 257 16 99054 99038 24.0813 13.375 0.0715938 0.0415115 2022-03-08T23:04:13.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 258 16 99277 99261 24.0419 13.9375 0.0386578 0.0415828 2022-03-08T23:04:13.180 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:13.178984 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:13.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:13.532799 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-08T23:04:14.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 259 16 99512 99496 24.0058 14.6875 0.136486 0.0416434 2022-03-08T23:04:14.485 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:14.482876 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:04:14.485 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:14.482900 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:15.056 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:15.054857 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:15.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:04:15.118004+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.041708 2022-03-08T23:04:15.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:04:15.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 260 16 99736 99720 23.9673 14 0.0165262 0.041708 2022-03-08T23:04:16.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 261 16 99924 99908 23.9205 11.75 0.0739974 0.0417939 2022-03-08T23:04:16.433 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:16.431618 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-08T23:04:16.434 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:16.431652 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T23:04:17.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 262 16 100134 100118 23.8793 13.125 0.0343151 0.0418669 2022-03-08T23:04:18.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 263 16 100351 100335 23.8401 13.5625 0.0349938 0.0419358 2022-03-08T23:04:18.296 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:18.292427 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T23:04:18.757 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:18.755528 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:18.774 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:18.772613 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:19.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 264 16 100581 100565 23.8042 14.375 0.0502778 0.0419957 2022-03-08T23:04:19.315 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:19.313394 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T23:04:20.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 265 16 100808 100792 23.7679 14.1875 0.0403754 0.0420625 2022-03-08T23:04:20.994 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:20.992924 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T23:04:21.119 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 266 16 101037 101021 23.7324 14.3125 0.141073 0.0421239 2022-03-08T23:04:21.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:21.231590 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:21.348 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:21.346622 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:04:21.349 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:21.346646 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:21.723 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:21.722039 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T23:04:22.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 267 16 101256 101240 23.6948 13.6875 0.0934041 0.0421898 2022-03-08T23:04:22.165 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:22.162640 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:23.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 268 16 101412 101396 23.6427 9.75 0.124333 0.0422763 2022-03-08T23:04:23.353 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:23.351972 I | op-mds: setting mds config flags 2022-03-08T23:04:23.354 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:23.351999 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:04:24.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 269 16 101627 101611 23.6048 13.4375 0.0297875 0.0423528 2022-03-08T23:04:24.235 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:24.230628 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:04:24.242 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:24.240671 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T23:04:25.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 270 16 101839 101823 23.5664 13.25 0.0953204 0.0424219 2022-03-08T23:04:25.565 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:25.560995 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:25.598 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:25.591510 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:25.945 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:25.939690 I | op-mds: setting mds config flags 2022-03-08T23:04:25.946 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:25.939719 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:04:26.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 271 16 102075 102059 23.5339 14.75 0.12801 0.0424807 2022-03-08T23:04:27.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 272 16 102272 102256 23.4926 12.3125 0.170886 0.0425521 2022-03-08T23:04:27.230 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:27.228037 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:04:28.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 273 16 102468 102452 23.4515 12.25 0.0333491 0.0426296 2022-03-08T23:04:28.717 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:28.715787 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:29.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 274 16 102695 102679 23.4176 14.1875 0.0752094 0.0426924 2022-03-08T23:04:30.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 275 16 102966 102950 23.3941 16.9375 0.0725165 0.0427355 2022-03-08T23:04:30.374 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:30.372356 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:04:30.374 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:30.372376 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:31.035 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:31.033449 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:31.120 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 276 16 103188 103172 23.3596 13.875 0.111507 0.0427923 2022-03-08T23:04:32.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 277 16 103359 103343 23.3138 10.6875 0.097437 0.0428801 2022-03-08T23:04:33.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 278 16 103576 103560 23.2788 13.5625 0.0320344 0.0429453 2022-03-08T23:04:34.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 279 16 103819 103803 23.2498 15.1875 0.0247779 0.0429977 2022-03-08T23:04:34.742 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:34.741031 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:34.790 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:34.788808 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:35.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:04:35.120002+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0430584 2022-03-08T23:04:35.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:04:35.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 280 16 104053 104037 23.2189 14.625 0.0457766 0.0430584 2022-03-08T23:04:36.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 281 16 104295 104279 23.1901 15.125 0.0376754 0.0431097 2022-03-08T23:04:37.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 282 16 104532 104516 23.1604 14.8125 0.0547234 0.0431658 2022-03-08T23:04:37.674 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:37.673193 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:38.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 283 16 104758 104742 23.1285 14.125 0.00960757 0.0432203 2022-03-08T23:04:38.679 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:38.675851 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:04:38.679 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:38.675880 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:39.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 284 16 104991 104975 23.0983 14.5625 0.0226496 0.0432808 2022-03-08T23:04:40.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 285 16 105263 105247 23.0769 17 0.00707603 0.0433191 2022-03-08T23:04:40.271 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:40.269755 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:41.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 286 16 105418 105402 23.0301 9.6875 0.0851203 0.0434048 2022-03-08T23:04:42.121 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 287 16 105644 105628 22.9991 14.125 0.0710655 0.0434632 2022-03-08T23:04:43.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 288 16 105845 105829 22.9628 12.5625 0.0268487 0.043538 2022-03-08T23:04:44.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 289 16 106094 106078 22.9372 15.5625 0.0651627 0.0435834 2022-03-08T23:04:45.005 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:45.003862 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:45.103 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:45.101672 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:45.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 290 16 106281 106265 22.8984 11.6875 0.191543 0.0436594 2022-03-08T23:04:46.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 291 16 106512 106496 22.8694 14.4375 0.0329863 0.043715 2022-03-08T23:04:47.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 292 16 106742 106726 22.8403 14.375 0.0612479 0.0437698 2022-03-08T23:04:47.559 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:47.557365 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:47.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:47.627786 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:04:47.634 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:47.627812 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:48.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 293 16 106963 106947 22.8095 13.8125 0.0480502 0.0438289 2022-03-08T23:04:48.269 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:48.267596 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:49.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 294 16 107163 107147 22.7744 12.5 0.119497 0.0438946 2022-03-08T23:04:50.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 295 16 107388 107372 22.7448 14.0625 0.0311975 0.0439524 2022-03-08T23:04:51.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 296 16 107620 107604 22.717 14.5 0.154047 0.044003 2022-03-08T23:04:51.501 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:51.498387 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:51.666 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:51.664390 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:52.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 297 16 107875 107859 22.6942 15.9375 0.106884 0.044051 2022-03-08T23:04:53.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 298 16 108113 108097 22.6679 14.875 0.0850345 0.0441046 2022-03-08T23:04:53.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:53.959431 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:04:54.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:54.117307 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:04:54.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:54.117331 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:04:54.122 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 299 16 108370 108354 22.6458 16.0625 0.0454431 0.0441469 2022-03-08T23:04:54.799 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:54.797572 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:04:55.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:04:55.121959+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0442187 2022-03-08T23:04:55.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:04:55.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 300 16 108548 108532 22.6074 11.125 0.0760184 0.0442187 2022-03-08T23:04:56.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 301 16 108729 108713 22.5699 11.3125 0.0684807 0.0442946 2022-03-08T23:04:57.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 302 16 108976 108960 22.5463 15.4375 0.0250477 0.0443425 2022-03-08T23:04:57.998 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:57.996295 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:04:58.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 303 16 109223 109207 22.5228 15.4375 0.0302002 0.0443888 2022-03-08T23:04:58.325 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:04:58.323732 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:04:59.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 304 16 109473 109457 22.5001 15.625 0.0611833 0.0444322 2022-03-08T23:05:00.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 305 16 109685 109669 22.4698 13.25 0.0316766 0.044485 2022-03-08T23:05:00.754 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:00.751802 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:00.948 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:00.946105 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:05:00.948 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:00.946131 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:01.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 306 16 109914 109898 22.4431 14.3125 0.0156946 0.044537 2022-03-08T23:05:01.707 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:01.703757 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:02.123 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 307 16 110137 110121 22.4154 13.9375 0.0278209 0.0445998 2022-03-08T23:05:03.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 308 16 110222 110206 22.3599 5.3125 0.0523925 0.0447092 2022-03-08T23:05:04.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 309 16 110243 110227 22.2918 1.3125 0.0235092 0.0447126 2022-03-08T23:05:05.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 310 16 110330 110314 22.2374 5.4375 0.112566 0.0449005 2022-03-08T23:05:05.516 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:05.513454 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:06.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 311 16 110494 110478 22.1989 10.25 0.0196014 0.0450372 2022-03-08T23:05:06.166 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:06.163850 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:05:07.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 312 16 110773 110757 22.1836 17.4375 0.0450674 0.0450684 2022-03-08T23:05:08.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 313 16 111054 111038 22.1688 17.5625 0.0519939 0.0450977 2022-03-08T23:05:08.461 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:08.459794 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:08.670 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:08.666253 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:05:08.671 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:08.666280 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:09.124 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 314 16 111323 111307 22.1518 16.8125 0.133141 0.0451227 2022-03-08T23:05:09.337 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:09.335695 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:10.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 315 16 111541 111525 22.1247 13.625 0.149254 0.0451794 2022-03-08T23:05:11.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 316 16 111845 111829 22.1148 19 0.0408011 0.0452082 2022-03-08T23:05:12.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 317 16 112186 112170 22.1123 21.3125 0.121769 0.0452125 2022-03-08T23:05:12.193 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:12.191856 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:13.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 318 16 112539 112523 22.1121 22.0625 0.0116616 0.0452137 2022-03-08T23:05:13.486 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:13.482548 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:05:14.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 319 16 112729 112713 22.08 11.875 0.0241735 0.0452081 2022-03-08T23:05:15.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:05:15.123997+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0453345 2022-03-08T23:05:15.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:05:15.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 320 16 112946 112930 22.0534 13.5625 0.0326918 0.0453345 2022-03-08T23:05:15.949 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:15.947258 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:16.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 321 16 113313 113297 22.0561 22.9375 0.0337394 0.0453293 2022-03-08T23:05:17.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 322 16 113576 113560 22.0387 16.4375 0.0462036 0.0453573 2022-03-08T23:05:17.494 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:17.490785 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:05:17.494 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:17.490813 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:18.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 323 16 113738 113722 22.0018 10.125 0.00871198 0.0453849 2022-03-08T23:05:18.169 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:18.147840 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:19.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 324 16 113738 113722 21.9339 0 - 0.0453849 2022-03-08T23:05:20.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 325 16 113966 113950 21.9102 7.125 0.0491307 0.0456293 2022-03-08T23:05:21.125 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 326 16 114243 114227 21.8961 17.3125 0.0567917 0.045659 2022-03-08T23:05:21.734 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:21.731478 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:22.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 327 16 114589 114573 21.8953 21.625 0.0322182 0.0456613 2022-03-08T23:05:23.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 328 16 114951 114935 21.8975 22.625 0.063729 0.0456562 2022-03-08T23:05:24.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 329 16 115261 115245 21.8898 19.375 0.215503 0.045669 2022-03-08T23:05:24.307 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:24.303738 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:05:25.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 330 16 115461 115445 21.8614 12.5 0.0359953 0.0457291 2022-03-08T23:05:25.930 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:25.926055 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:26.083 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:26.081372 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:05:26.083 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:26.081395 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:26.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 331 16 115804 115788 21.8601 21.4375 0.0657053 0.0457357 2022-03-08T23:05:26.663 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:26.660927 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:27.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 332 16 116067 116051 21.8438 16.4375 0.175054 0.0457642 2022-03-08T23:05:28.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 333 16 116316 116300 21.8249 15.5625 0.148283 0.0458073 2022-03-08T23:05:29.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 334 16 116644 116628 21.8209 20.5 0.115385 0.045817 2022-03-08T23:05:29.591 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:29.588260 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:30.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 335 16 117000 116984 21.8222 22.25 0.106543 0.045816 2022-03-08T23:05:31.126 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 336 16 117324 117308 21.8175 20.25 0.100877 0.0458237 2022-03-08T23:05:32.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 337 16 117711 117695 21.8245 24.1875 0.0556902 0.0458114 2022-03-08T23:05:33.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 338 16 118077 118061 21.8276 22.875 0.0247257 0.0458029 2022-03-08T23:05:34.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 339 16 118364 118348 21.8161 17.9375 0.052863 0.0457983 2022-03-08T23:05:34.865 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:34.863830 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:05:35.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:05:35.126016+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0458932 2022-03-08T23:05:35.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:05:35.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 340 16 118539 118523 21.7842 10.9375 0.00963662 0.0458932 2022-03-08T23:05:36.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 341 16 118873 118857 21.7815 20.875 0.0123796 0.0458997 2022-03-08T23:05:37.086 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:37.083905 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:37.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 342 16 119196 119180 21.7768 20.1875 0.0234994 0.0459107 2022-03-08T23:05:37.205 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:37.203345 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:05:37.205 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:37.203369 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:37.749 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:37.747514 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:38.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 343 16 119458 119442 21.7611 16.375 0.0335795 0.0459384 2022-03-08T23:05:39.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 344 16 119708 119692 21.7432 15.625 0.0156101 0.0459817 2022-03-08T23:05:40.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 345 16 120050 120034 21.7421 21.375 0.0102442 0.0459845 2022-03-08T23:05:40.921 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:40.917582 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:05:41.127 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 346 16 120368 120352 21.7367 19.875 0.0314397 0.0459933 2022-03-08T23:05:42.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 347 16 120741 120725 21.7413 23.3125 0.0475534 0.0459859 2022-03-08T23:05:43.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 348 16 121103 121087 21.7438 22.625 0.0799823 0.04598 2022-03-08T23:05:44.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 349 16 121482 121466 21.7494 23.6875 0.0376331 0.0459677 2022-03-08T23:05:45.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 350 16 121535 121519 21.6967 3.3125 0.0264806 0.045966 2022-03-08T23:05:46.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 351 16 121769 121753 21.6765 14.625 0.0218916 0.0461218 2022-03-08T23:05:47.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 352 16 122110 122094 21.6755 21.3125 0.0331601 0.0461244 2022-03-08T23:05:48.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 353 16 122472 122456 21.6782 22.625 0.0281385 0.0461205 2022-03-08T23:05:49.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 354 16 122778 122762 21.671 19.125 0.0326652 0.0461351 2022-03-08T23:05:50.128 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 355 16 122972 122956 21.6441 12.125 0.0200876 0.0461344 2022-03-08T23:05:51.129 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 356 16 123317 123301 21.6438 21.5625 0.0134383 0.0461939 2022-03-08T23:05:51.168 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:51.165933 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:05:52.129 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 357 16 123651 123635 21.6416 20.875 0.0614273 0.0461968 2022-03-08T23:05:53.129 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 358 16 123965 123949 21.636 19.625 0.100047 0.0462088 2022-03-08T23:05:53.401 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:53.399079 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:05:53.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:53.463062 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:05:53.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:53.463119 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:05:54.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:54.029906 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:05:54.129 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 359 16 124305 124289 21.6349 21.25 0.0728398 0.0462092 2022-03-08T23:05:55.130 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:2022-03-08T23:05:55.128738+0000 min lat: 0.00118928 max lat: 12.8523 avg lat: 0.0462742 2022-03-08T23:05:55.130 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-08T23:05:55.130 INFO:tasks.radosbench.radosbench.a.smithi176.stdout: 360 3 124468 124465 21.6054 11 0.068189 0.0462742 2022-03-08T23:05:56.130 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Total time run: 360.071 2022-03-08T23:05:56.130 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Total writes made: 124468 2022-03-08T23:05:56.131 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Write size: 65536 2022-03-08T23:05:56.131 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Object size: 65536 2022-03-08T23:05:56.131 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Bandwidth (MB/sec): 21.6047 2022-03-08T23:05:56.131 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Stddev Bandwidth: 13.9974 2022-03-08T23:05:56.131 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Max bandwidth (MB/sec): 87.25 2022-03-08T23:05:56.132 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Min bandwidth (MB/sec): 0 2022-03-08T23:05:56.132 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Average IOPS: 345 2022-03-08T23:05:56.132 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Stddev IOPS: 223.967 2022-03-08T23:05:56.132 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Max IOPS: 1396 2022-03-08T23:05:56.133 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Min IOPS: 0 2022-03-08T23:05:56.133 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Average Latency(s): 0.0462754 2022-03-08T23:05:56.133 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Stddev Latency(s): 0.211571 2022-03-08T23:05:56.133 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Max latency(s): 12.8523 2022-03-08T23:05:56.134 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Min latency(s): 0.00118928 2022-03-08T23:05:56.537 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Cleaning up (deleting benchmark objects) 2022-03-08T23:05:57.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:05:57.631161 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:06:18.126 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:06:18.122650 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:06:20.166 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:06:20.164646 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:06:20.297 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:06:20.295248 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:06:20.298 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:06:20.295275 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:06:21.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:06:21.329187 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:06:24.735 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:06:24.733840 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:07:05.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:07:05.705201 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:07:07.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:07:07.835692 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:07:08.016 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:07:08.014422 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:07:08.016 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:07:08.014443 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:07:08.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:07:08.544860 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:07:11.869 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:07:11.866450 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:08:34.482 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:08:34.475049 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:08:36.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:08:36.564354 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:08:38.797 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:08:38.793707 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:08:38.798 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:08:38.793733 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:08:39.470 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:08:39.466537 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:08:42.637 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:08:42.634937 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:11:26.485 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:11:26.481748 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:11:28.816 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:11:28.814473 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:11:28.890 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:11:28.889066 I | ceph-object-controller: ceph object store gateway service running at 10.109.162.68 2022-03-08T23:11:28.891 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:11:28.889086 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:11:29.637 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:11:29.634542 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:11:32.879 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:11:32.877543 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-08T23:12:03.456 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Removed 124468 objects 2022-03-08T23:12:03.457 INFO:tasks.radosbench.radosbench.a.smithi176.stdout:Clean up completed and total clean up time :366.92 2022-03-08T23:12:07.683 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-08T23:12:07.880 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-08T23:12:11.488 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'unique_pool_0' removed 2022-03-08T23:12:11.514 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-08T23:12:11.526 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-08T23:12:11.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-08T23:12:11.538 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T23:12:11.703 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:12:12.731 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:12:12.731 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:03:47.000000Z", "last_refresh": "2022-03-08T23:12:12.627792Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:12.627792Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:12:12.627792Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:12:12.627792Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:12:12.627792Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:12.627792Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T23:12:12.627792Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:12.627792Z", "running": 0, "size": 1}}] 2022-03-08T23:12:12.751 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch rm mds.foo 2022-03-08T23:12:13.271 INFO:teuthology.orchestra.run.smithi176.stdout:Removed cephfilesystems/foo 2022-03-08T23:12:13.280 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:13.279316 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:12:13.292 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch rm rbd-mirror 2022-03-08T23:12:13.906 INFO:teuthology.orchestra.run.smithi176.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T23:12:13.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:13.929725 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T23:12:13.941 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch rm rgw.foo 2022-03-08T23:12:14.700 INFO:teuthology.orchestra.run.smithi176.stdout:Removed cephobjectstores/foo 2022-03-08T23:12:14.726 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:12:15.048 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:15.046907 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:12:15.072 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:15.070257 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:16.721 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:16.676627 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:16.722 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:16.676653 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T23:12:16.722 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:16.676659 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T23:12:16.963 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:12:16.963 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:03:47.000000Z", "last_refresh": "2022-03-08T23:12:16.191620Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:16.191620Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:12:16.191620Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:12:16.191620Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:12:16.191620Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:16.191620Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:16.191620Z", "running": 0, "size": 1}}] 2022-03-08T23:12:18.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:18.329270 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T23:12:19.442 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:19.340268 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:19.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:19.414843 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:19.405+0000 7f546b676340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:19.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:12:19.405+0000 7f546b676340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:19.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:12:19.405+0000 7f546b676340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:19.443 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:19.444 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:19.414867 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:12:19.405+0000 7f546b676340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:12:19.444 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:12:19.405+0000 7f546b676340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:12:19.444 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:12:19.405+0000 7f546b676340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:12:19.444 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:12:21.580 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:21.578846 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:12:21.580 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:21.578867 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T23:12:21.581 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:21.578872 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T23:12:22.374 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:22.372483 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T23:12:22.374 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:22.372509 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T23:12:23.573 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:23.571928 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T23:12:26.985 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:12:28.344 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:12:28.345 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:03:47.000000Z", "last_refresh": "2022-03-08T23:12:27.675277Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:27.675277Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:12:27.675277Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:12:27.675277Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:12:27.675277Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:27.675277Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:27.675277Z", "running": 0, "size": 1}}] 2022-03-08T23:12:29.377 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:29.373159 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T23:12:29.377 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:29.373185 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T23:12:31.638 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:31.636231 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T23:12:32.065 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:32.063291 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T23:12:32.065 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:32.063324 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T23:12:34.533 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:34.531625 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T23:12:34.533 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:34.531681 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:12:34.606 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:34.605102 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T23:12:34.611 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:34.609796 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T23:12:34.613 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:34.610580 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T23:12:34.614 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:34.610657 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T23:12:38.362 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:12:38.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:12:38.959339 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi176": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:12:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:12:39.341 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:03:47.000000Z", "last_refresh": "2022-03-08T23:12:38.975989Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:12:38.975989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:12:38.975989Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:12:38.975989Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:38.975989Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:38.975989Z", "running": 0, "size": 1}}] 2022-03-08T23:12:49.364 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:12:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:12:50.917 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:03:47.000000Z", "last_refresh": "2022-03-08T23:12:49.876431Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:12:49.876431Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:12:49.876431Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:12:49.876431Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:12:49.876431Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:12:49.876431Z", "running": 0, "size": 1}}] 2022-03-08T23:13:00.937 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:13:01.561 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:13:01.561 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:13:01.470935Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:13:01.470935Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:13:01.470935Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:13:01.470935Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:01.470935Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:01.470935Z", "running": 0, "size": 1}}] 2022-03-08T23:13:11.584 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:13:12.202 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:13:12.203 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:13:12.099952Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:13:12.099952Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:13:12.099952Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:13:12.099952Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:12.099952Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:12.099952Z", "running": 0, "size": 1}}] 2022-03-08T23:13:22.227 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:13:23.845 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:13:23.846 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:13:22.754854Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:13:22.754854Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:13:22.754854Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:13:22.754854Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:22.754854Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:22.754854Z", "running": 0, "size": 1}}] 2022-03-08T23:13:33.867 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:13:34.473 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:13:34.473 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:13:34.387776Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:13:34.387776Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:13:34.387776Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:13:34.387776Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:34.387776Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:34.387776Z", "running": 0, "size": 1}}] 2022-03-08T23:13:44.496 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:13:45.160 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:13:45.161 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:13:45.017317Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:13:45.017317Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:13:45.017317Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:13:45.017317Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:45.017317Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:45.017317Z", "running": 0, "size": 1}}] 2022-03-08T23:13:55.185 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:13:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:13:56.068 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:13:55.959323Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:13:55.959323Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:13:55.959323Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:13:55.959323Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:13:55.959323Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:13:55.959323Z", "running": 0, "size": 1}}] 2022-03-08T23:14:06.093 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:14:06.705 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:14:06.705 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:14:06.613921Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:14:06.613921Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:14:06.613921Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:14:06.613921Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:06.613921Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:06.613921Z", "running": 0, "size": 1}}] 2022-03-08T23:14:16.728 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:14:17.351 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:14:17.352 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:14:17.251107Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:14:17.251107Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:14:17.251107Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:14:17.251107Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:17.251107Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:17.251107Z", "running": 0, "size": 1}}] 2022-03-08T23:14:27.377 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:14:27.970 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:14:27.971 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:14:27.887160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:14:27.887160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:14:27.887160Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:14:27.887160Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:27.887160Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:27.887160Z", "running": 0, "size": 1}}] 2022-03-08T23:14:37.995 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:14:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:14:38.607 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:14:38.501934Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:14:38.501934Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:14:38.501934Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:14:38.501934Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:38.501934Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:38.501934Z", "running": 0, "size": 1}}] 2022-03-08T23:14:48.629 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:14:49.270 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:14:49.270 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:14:49.156737Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:14:49.156737Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:14:49.156737Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:14:49.156737Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:49.156737Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:49.156737Z", "running": 0, "size": 1}}] 2022-03-08T23:14:59.294 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:14:59.951 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:14:59.952 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:14:59.833611Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:14:59.833611Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:14:59.833611Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:14:59.833611Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:14:59.833611Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:14:59.833611Z", "running": 0, "size": 1}}] 2022-03-08T23:15:09.977 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:15:10.769 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:15:10.769 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:15:10.524434Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:15:10.524434Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:15:10.524434Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:15:10.524434Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:10.524434Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:10.524434Z", "running": 0, "size": 1}}] 2022-03-08T23:15:20.793 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:15:21.407 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:15:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:15:21.310088Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:15:21.310088Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:15:21.310088Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:15:21.310088Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:21.310088Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:21.310088Z", "running": 0, "size": 1}}] 2022-03-08T23:15:31.433 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:15:32.074 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:15:32.075 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:15:31.960345Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:15:31.960345Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:15:31.960345Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:15:31.960345Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:31.960345Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:31.960345Z", "running": 0, "size": 1}}] 2022-03-08T23:15:42.096 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:15:42.689 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:15:42.689 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:15:42.593578Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:15:42.593578Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:15:42.593578Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:15:42.593578Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:42.593578Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:42.593578Z", "running": 0, "size": 1}}] 2022-03-08T23:15:52.716 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:15:53.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:15:53.323 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:15:53.224678Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:15:53.224678Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:15:53.224678Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:15:53.224678Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:53.224678Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:15:53.224678Z", "running": 0, "size": 1}}] 2022-03-08T23:16:03.349 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:16:03.952 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:16:03.952 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:16:03.860504Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:16:03.860504Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:16:03.860504Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:16:03.860504Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:03.860504Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:03.860504Z", "running": 0, "size": 1}}] 2022-03-08T23:16:13.978 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:16:14.937 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:16:14.937 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:16:14.539223Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:16:14.539223Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:16:14.539223Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:16:14.539223Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:14.539223Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:14.539223Z", "running": 0, "size": 1}}] 2022-03-08T23:16:24.959 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:16:25.547 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:16:25.547 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:16:25.447424Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:16:25.447424Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:16:25.447424Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:16:25.447424Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:25.447424Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:25.447424Z", "running": 0, "size": 1}}] 2022-03-08T23:16:35.570 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:16:36.225 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:16:36.225 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:16:36.129085Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:16:36.129085Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:16:36.129085Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:16:36.129085Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:36.129085Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:36.129085Z", "running": 0, "size": 1}}] 2022-03-08T23:16:46.247 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:16:46.890 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:16:46.890 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:16:46.791458Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:16:46.791458Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:16:46.791458Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:16:46.791458Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:46.791458Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:46.791458Z", "running": 0, "size": 1}}] 2022-03-08T23:16:56.913 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:16:57.543 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:16:57.544 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:16:57.425817Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:16:57.425817Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:16:57.425817Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:16:57.425817Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:57.425817Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:16:57.425817Z", "running": 0, "size": 1}}] 2022-03-08T23:17:00.565 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:17:00.564282 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:17:01.176 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:17:01.174401 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:17:01.167+0000 7f83da6d6340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:17:01.176 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:17:01.167+0000 7f83da6d6340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:17:01.177 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:17:01.167+0000 7f83da6d6340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:17:01.177 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:17:01.177 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:17:01.174426 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:17:01.167+0000 7f83da6d6340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:17:01.178 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:17:01.167+0000 7f83da6d6340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:17:01.178 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:17:01.167+0000 7f83da6d6340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:17:01.178 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:17:07.569 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:17:08.156 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:17:08.156 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:17:08.053188Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:17:08.053188Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:17:08.053188Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:17:08.053188Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:08.053188Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:08.053188Z", "running": 0, "size": 1}}] 2022-03-08T23:17:18.182 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:17:18.831 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:17:18.832 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:17:18.712843Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:17:18.712843Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:17:18.712843Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:17:18.712843Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:18.712843Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:18.712843Z", "running": 0, "size": 1}}] 2022-03-08T23:17:28.860 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:17:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:17:29.477 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:17:29.381425Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:17:29.381425Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:17:29.381425Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:17:29.381425Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:29.381425Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:29.381425Z", "running": 0, "size": 1}}] 2022-03-08T23:17:39.500 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:17:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:17:40.159 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:17:40.059970Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:17:40.059970Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:17:40.059970Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:17:40.059970Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:40.059970Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:40.059970Z", "running": 0, "size": 1}}] 2022-03-08T23:17:50.184 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:17:50.828 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:17:50.829 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:17:50.719385Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:17:50.719385Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:17:50.719385Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:17:50.719385Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:50.719385Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:17:50.719385Z", "running": 0, "size": 1}}] 2022-03-08T23:18:00.854 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:18:01.467 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:18:01.467 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:18:01.347646Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:18:01.347646Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:18:01.347646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:18:01.347646Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:01.347646Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:01.347646Z", "running": 0, "size": 1}}] 2022-03-08T23:18:11.492 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:18:12.050325Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:18:12.050325Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:18:12.050325Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:18:12.050325Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:12.050325Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:12.050325Z", "running": 0, "size": 1}}] 2022-03-08T23:18:22.209 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:18:22.831 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:18:22.832 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:18:22.736643Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:18:22.736643Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:18:22.736643Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:18:22.736643Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:22.736643Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:22.736643Z", "running": 0, "size": 1}}] 2022-03-08T23:18:32.856 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:18:33.464 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:18:33.464 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:18:33.365741Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:18:33.365741Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:18:33.365741Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:18:33.365741Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:33.365741Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:33.365741Z", "running": 0, "size": 1}}] 2022-03-08T23:18:43.488 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:18:44.157 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:18:44.157 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:18:44.035834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:18:44.035834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:18:44.035834Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:18:44.035834Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:44.035834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:44.035834Z", "running": 0, "size": 1}}] 2022-03-08T23:18:54.180 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:18:54.796 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:18:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:18:54.697184Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:18:54.697184Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:18:54.697184Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:18:54.697184Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:54.697184Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:18:54.697184Z", "running": 0, "size": 1}}] 2022-03-08T23:19:04.817 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:19:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:19:05.470 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:19:05.329506Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:19:05.329506Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:19:05.329506Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:19:05.329506Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:05.329506Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:05.329506Z", "running": 0, "size": 1}}] 2022-03-08T23:19:15.499 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:19:16.246 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:19:16.246 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:19:16.152407Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:19:16.152407Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:19:16.152407Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:19:16.152407Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:16.152407Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:16.152407Z", "running": 0, "size": 1}}] 2022-03-08T23:19:26.266 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:19:26.888 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:19:26.888 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:19:26.806497Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:19:26.806497Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:19:26.806497Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:19:26.806497Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:26.806497Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:26.806497Z", "running": 0, "size": 1}}] 2022-03-08T23:19:36.910 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:19:37.566 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:19:37.566 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:19:37.452568Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:19:37.452568Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:19:37.452568Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:19:37.452568Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:37.452568Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:37.452568Z", "running": 0, "size": 1}}] 2022-03-08T23:19:47.589 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:19:48.227 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:19:48.228 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:19:48.135063Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:19:48.135063Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:19:48.135063Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:19:48.135063Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:48.135063Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:48.135063Z", "running": 0, "size": 1}}] 2022-03-08T23:19:58.251 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:19:58.865 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:19:58.865 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:19:58.745743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:19:58.745743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:19:58.745743Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:19:58.745743Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:58.745743Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:19:58.745743Z", "running": 0, "size": 1}}] 2022-03-08T23:20:08.883 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:20:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:20:09.491 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:20:09.389412Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:20:09.389412Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:20:09.389412Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:20:09.389412Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:09.389412Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:09.389412Z", "running": 0, "size": 1}}] 2022-03-08T23:20:19.516 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:20:20.180 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:20:20.180 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:20:20.077339Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:20:20.077339Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:20:20.077339Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:20:20.077339Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:20.077339Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:20.077339Z", "running": 0, "size": 1}}] 2022-03-08T23:20:30.201 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:20:30.858 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:20:30.858 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:20:30.760735Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:20:30.760735Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:20:30.760735Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:20:30.760735Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:30.760735Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:30.760735Z", "running": 0, "size": 1}}] 2022-03-08T23:20:40.884 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:20:41.520 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:20:41.520 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:20:41.424714Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:20:41.424714Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:20:41.424714Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:20:41.424714Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:41.424714Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:41.424714Z", "running": 0, "size": 1}}] 2022-03-08T23:20:51.546 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:20:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:20:52.207 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:20:52.110728Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:20:52.110728Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:20:52.110728Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:20:52.110728Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:52.110728Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:20:52.110728Z", "running": 0, "size": 1}}] 2022-03-08T23:21:02.233 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:21:02.871 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:21:02.871 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:21:02.757404Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:21:02.757404Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:21:02.757404Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:21:02.757404Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:02.757404Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:02.757404Z", "running": 0, "size": 1}}] 2022-03-08T23:21:12.894 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:21:13.521 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:21:13.522 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:21:13.418881Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:21:13.418881Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:21:13.418881Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:21:13.418881Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:13.418881Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:13.418881Z", "running": 0, "size": 1}}] 2022-03-08T23:21:23.545 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:21:24.273 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:21:24.274 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:21:24.089398Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:21:24.089398Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:21:24.089398Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:21:24.089398Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:24.089398Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:24.089398Z", "running": 0, "size": 1}}] 2022-03-08T23:21:34.315 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:21:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:21:34.900 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:21:34.803033Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:21:34.803033Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:21:34.803033Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:21:34.803033Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:34.803033Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:34.803033Z", "running": 0, "size": 1}}] 2022-03-08T23:21:44.923 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:21:45.609 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:21:45.609 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:21:45.460743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:21:45.460743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:21:45.460743Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:21:45.460743Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:45.460743Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:45.460743Z", "running": 0, "size": 1}}] 2022-03-08T23:21:55.633 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:21:56.253 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:21:56.254 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:21:56.154087Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:21:56.154087Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:21:56.154087Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:21:56.154087Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:56.154087Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:21:56.154087Z", "running": 0, "size": 1}}] 2022-03-08T23:22:06.279 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:22:06.920 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:22:06.920 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:22:06.819806Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:22:06.819806Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:22:06.819806Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:22:06.819806Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:06.819806Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:06.819806Z", "running": 0, "size": 1}}] 2022-03-08T23:22:16.942 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:22:20.310 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:22:20.310 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:22:20.211784Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:22:20.211784Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:22:20.211784Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:22:20.211784Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:20.211784Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:20.211784Z", "running": 0, "size": 1}}] 2022-03-08T23:22:30.330 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:22:31.075 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:22:31.076 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:22:30.929842Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:22:30.929842Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:22:30.929842Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:22:30.929842Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:30.929842Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:30.929842Z", "running": 0, "size": 1}}] 2022-03-08T23:22:41.101 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:22:41.843 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:22:41.844 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:22:41.704016Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:22:41.704016Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:22:41.704016Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:22:41.704016Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:41.704016Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:41.704016Z", "running": 0, "size": 1}}] 2022-03-08T23:22:51.866 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:22:52.457 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:22:52.457 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:22:52.370270Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:22:52.370270Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:22:52.370270Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:22:52.370270Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:52.370270Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:22:52.370270Z", "running": 0, "size": 1}}] 2022-03-08T23:23:02.481 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:23:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:23:03.170 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:23:03.042994Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:23:03.042994Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:23:03.042994Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:23:03.042994Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:03.042994Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:03.042994Z", "running": 0, "size": 1}}] 2022-03-08T23:23:13.195 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:23:13.812 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:23:13.813 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:23:13.714315Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:23:13.714315Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:23:13.714315Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:23:13.714315Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:13.714315Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:13.714315Z", "running": 0, "size": 1}}] 2022-03-08T23:23:23.835 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:23:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:23:24.518 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:23:24.423616Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:23:24.423616Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:23:24.423616Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:23:24.423616Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:24.423616Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:24.423616Z", "running": 0, "size": 1}}] 2022-03-08T23:23:34.541 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:23:35.161 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:23:35.161 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:23:35.068705Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:23:35.068705Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:23:35.068705Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:23:35.068705Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:35.068705Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:35.068705Z", "running": 0, "size": 1}}] 2022-03-08T23:23:45.185 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:23:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:23:45.788 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:23:45.670770Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:23:45.670770Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:23:45.670770Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:23:45.670770Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:45.670770Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:45.670770Z", "running": 0, "size": 1}}] 2022-03-08T23:23:55.817 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:23:56.554 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:23:56.554 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:23:56.461194Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:23:56.461194Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:23:56.461194Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:23:56.461194Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:56.461194Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:23:56.461194Z", "running": 0, "size": 1}}] 2022-03-08T23:24:06.582 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:24:07.212 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:24:07.212 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:24:07.105695Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:24:07.105695Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:24:07.105695Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:24:07.105695Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:07.105695Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:07.105695Z", "running": 0, "size": 1}}] 2022-03-08T23:24:17.239 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:24:17.865 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:24:17.865 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:24:17.763651Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:24:17.763651Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:24:17.763651Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:24:17.763651Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:17.763651Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:17.763651Z", "running": 0, "size": 1}}] 2022-03-08T23:24:27.889 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:24:28.481 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:24:28.482 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:24:28.386141Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:24:28.386141Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:24:28.386141Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:24:28.386141Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:28.386141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:28.386141Z", "running": 0, "size": 1}}] 2022-03-08T23:24:38.505 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:24:39.133 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:24:39.133 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:24:39.038369Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:24:39.038369Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:24:39.038369Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:24:39.038369Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:39.038369Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:39.038369Z", "running": 0, "size": 1}}] 2022-03-08T23:24:49.156 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:24:49.694632Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:24:49.694632Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:24:49.694632Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:24:49.694632Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:49.694632Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:24:49.694632Z", "running": 0, "size": 1}}] 2022-03-08T23:24:59.820 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:25:00.417 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:25:00.417 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:25:00.327614Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:25:00.327614Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:25:00.327614Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:25:00.327614Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:00.327614Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:00.327614Z", "running": 0, "size": 1}}] 2022-03-08T23:25:10.437 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:25:11.188 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:25:11.188 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:25:11.027683Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:25:11.027683Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:25:11.027683Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:25:11.027683Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:11.027683Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:11.027683Z", "running": 0, "size": 1}}] 2022-03-08T23:25:21.210 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:25:21.799 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:25:21.800 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:25:21.705346Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:25:21.705346Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:25:21.705346Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:25:21.705346Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:21.705346Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:21.705346Z", "running": 0, "size": 1}}] 2022-03-08T23:25:31.822 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:25:32.718 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:25:32.718 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:25:32.620375Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:25:32.620375Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:25:32.620375Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:25:32.620375Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:32.620375Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:32.620375Z", "running": 0, "size": 1}}] 2022-03-08T23:25:42.746 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:25:43.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:25:43.372 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:25:43.265467Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:25:43.265467Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:25:43.265467Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:25:43.265467Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:43.265467Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:43.265467Z", "running": 0, "size": 1}}] 2022-03-08T23:25:53.394 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:25:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:25:54.030 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:25:53.938670Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:25:53.938670Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:25:53.938670Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:25:53.938670Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:53.938670Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:25:53.938670Z", "running": 0, "size": 1}}] 2022-03-08T23:26:04.057 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:26:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:26:04.665 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:26:04.585117Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:26:04.585117Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:26:04.585117Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:26:04.585117Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:04.585117Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:04.585117Z", "running": 0, "size": 1}}] 2022-03-08T23:26:14.688 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:26:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:26:15.321 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:26:15.225003Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:26:15.225003Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:26:15.225003Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:26:15.225003Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:15.225003Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:15.225003Z", "running": 0, "size": 1}}] 2022-03-08T23:26:25.345 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:26:25.956 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:26:25.957 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:26:25.860496Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:26:25.860496Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:26:25.860496Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:26:25.860496Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:25.860496Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:25.860496Z", "running": 0, "size": 1}}] 2022-03-08T23:26:35.981 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:26:36.594 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:26:36.594 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:26:36.500935Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:26:36.500935Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:26:36.500935Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:26:36.500935Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:36.500935Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:36.500935Z", "running": 0, "size": 1}}] 2022-03-08T23:26:46.616 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:26:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:26:47.221 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:26:47.135281Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:26:47.135281Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:26:47.135281Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:26:47.135281Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:47.135281Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:47.135281Z", "running": 0, "size": 1}}] 2022-03-08T23:26:57.244 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:26:57.905 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:26:57.905 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:26:57.786058Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:26:57.786058Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:26:57.786058Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:26:57.786058Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:57.786058Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:26:57.786058Z", "running": 0, "size": 1}}] 2022-03-08T23:27:07.930 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:27:08.562 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:27:08.562 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:27:08.471672Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:27:08.471672Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:27:08.471672Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:27:08.471672Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:08.471672Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:08.471672Z", "running": 0, "size": 1}}] 2022-03-08T23:27:18.585 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:27:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:27:19.258 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:27:19.126731Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:27:19.126731Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:27:19.126731Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:27:19.126731Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:19.126731Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:19.126731Z", "running": 0, "size": 1}}] 2022-03-08T23:27:29.278 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:27:29.932 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:27:29.933 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:27:29.818340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:27:29.818340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:27:29.818340Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:27:29.818340Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:29.818340Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:29.818340Z", "running": 0, "size": 1}}] 2022-03-08T23:27:39.955 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:27:40.598 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:27:40.599 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:27:40.494644Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:27:40.494644Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:27:40.494644Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:27:40.494644Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:40.494644Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:40.494644Z", "running": 0, "size": 1}}] 2022-03-08T23:27:50.622 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:27:51.253 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:27:51.253 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:27:51.164019Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:27:51.164019Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:27:51.164019Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:27:51.164019Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:51.164019Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:27:51.164019Z", "running": 0, "size": 1}}] 2022-03-08T23:28:01.278 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:28:01.914 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:28:01.914 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:28:01.821358Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:28:01.821358Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:28:01.821358Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:28:01.821358Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:01.821358Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:28:01.821358Z", "running": 0, "size": 1}}] 2022-03-08T23:28:11.938 DEBUG:teuthology.orchestra.run.smithi176:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-4qgtx -- ceph orch ls -f json 2022-03-08T23:28:12.614 INFO:teuthology.orchestra.run.smithi176.stdout: 2022-03-08T23:28:12.615 INFO:teuthology.orchestra.run.smithi176.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T23:12:39.000000Z", "last_refresh": "2022-03-08T23:28:12.518303Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:55:25.000000Z", "last_refresh": "2022-03-08T23:28:12.518303Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:54:53.000000Z", "last_refresh": "2022-03-08T23:28:12.518303Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-08T22:56:50.000000Z", "last_refresh": "2022-03-08T23:28:12.518303Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:12.518303Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-08T23:28:12.518303Z", "running": 0, "size": 1}}] 2022-03-08T23:28:12.639 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:12.640 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:12.640 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T23:28:12.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:28:12.692 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:12.693 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:12.712 DEBUG:teuthology.orchestra.remote:smithi176:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-08T23:28:12.723 DEBUG:teuthology.orchestra.run.smithi176:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-08T23:28:12.784 INFO:teuthology.orchestra.run.smithi176.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T23:28:12.799 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:28:12.799 DEBUG:teuthology.orchestra.run.smithi176:> kubectl delete -f cluster.yaml 2022-03-08T23:28:12.948 INFO:teuthology.orchestra.run.smithi176.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T23:28:12.949 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:12.948975 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T23:28:13.261 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:13.261183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:13.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:13.268716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:13.269 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:13.268748 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:23.490 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:23.490253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:23.503 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:23.503454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:33.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:33.723770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:33.736 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:33.736395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:43.958 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:43.958172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:43.969 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:43.969769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:54.192 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:54.192207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:28:54.204 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:28:54.203922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:04.458 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:04.457544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:04.474 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:04.474092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:14.695 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:14.695388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:14.718 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:14.718461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:24.952 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:24.952545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:24.965 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:24.963959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:35.185 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:35.185591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:35.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:35.198476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:45.417 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:45.416737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:45.430 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:45.429216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:55.656 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:55.656280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:29:55.668 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:29:55.668693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:05.887 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:05.887188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:05.901 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:05.900755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:16.125 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:16.124697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:16.140 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:16.140048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:26.360 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:26.360541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:26.373 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:26.373430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:36.599 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:36.598869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:36.613 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:36.613142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:46.840 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:46.839707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:46.851 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:46.851197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:57.087 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:57.087035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:30:57.095 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:30:57.095522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:07.314 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:07.314065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:07.324 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:07.323968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:17.536 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:17.536386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:17.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:17.548266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:17.962 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:17.962125 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:31:17.949859074 +0000 UTC m=+2243.939224210 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:31:27.768 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:27.768089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:27.780 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:27.780260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:38.031 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:38.030887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:38.045 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:38.044163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:48.271 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:48.271630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:48.292 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:48.292614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:58.512 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:58.512229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:31:58.550 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:31:58.550109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:08.771 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:08.770980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:08.782 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:08.782480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:18.813 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:18.813739 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:32:18.805639311 +0000 UTC m=+2304.795004404 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:32:18.996 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:18.996607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:19.009 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:19.009403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:29.230 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:29.230413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:29.244 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:29.244722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:39.468 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:39.468304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:39.480 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:39.480638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:49.701 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:49.701140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:49.709 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:49.709381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:59.982 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:59.982424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:32:59.991 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:32:59.991164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:10.240 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:10.240436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:10.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:10.250841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:20.489 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:20.488798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:20.498 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:20.498481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:30.919 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:30.919373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:30.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:30.932304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:41.150 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:41.149968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:41.161 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:41.161106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:41.202 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:41.202686 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:33:41.804 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:41.803450 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:33:41.794+0000 7f8aaf013340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:33:41.804 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:33:41.794+0000 7f8aaf013340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:33:41.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:33:41.794+0000 7f8aaf013340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:33:41.805 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:33:41.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:41.803473 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:33:41.794+0000 7f8aaf013340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:33:41.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:33:41.794+0000 7f8aaf013340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:33:41.806 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:33:41.794+0000 7f8aaf013340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:33:41.806 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:33:51.380 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:51.379902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:33:51.392 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:33:51.392382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:01.648 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:01.648622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:01.661 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:01.661570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:11.902 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:11.902407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:11.913 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:11.914166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:22.179 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:22.178804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:22.194 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:22.190916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:32.408 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:32.407736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:32.420 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:32.420284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:42.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:42.649699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:42.661 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:42.661989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:52.880 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:52.880634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:34:52.891 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:34:52.891226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:03.137 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:03.137720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:03.148 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:03.148984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:13.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:13.367757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:13.380 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:13.380413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:23.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:23.601133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:23.613 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:23.614381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:33.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:33.900477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:33.913 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:33.913637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:44.130 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:44.130087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:44.141 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:44.140447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:54.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:54.357195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:35:54.373 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:35:54.373427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:04.719 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:04.719012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:04.730 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:04.730963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:14.952 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:14.952617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:14.962 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:14.963067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:25.181 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:25.181482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:25.262 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:25.262098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:35.515 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:35.515322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:35.523 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:35.524052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:45.788 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:45.788369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:45.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:45.801477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:56.019 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:56.018919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:36:56.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:36:56.032224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:06.387 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:06.387790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:06.404 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:06.402734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:16.627 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:16.626923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:16.635 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:16.635199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:26.856 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:26.856433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:26.864 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:26.864471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:37.081 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:37.081790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:37.094 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:37.094607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:47.312 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:47.312501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:47.323 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:47.324044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:57.544 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:57.543986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:37:57.555 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:37:57.555444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:07.866 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:07.866745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:07.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:07.875751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:18.095 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:18.095172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:18.106 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:18.107120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:28.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:28.330976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:28.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:28.343809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:38.699 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:38.699095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:38.712 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:38.711552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:48.969 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:48.969828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:48.980 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:48.980474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:59.228 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:59.228171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:38:59.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:38:59.237875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:09.691 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:09.690533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:09.717 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:09.717217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:19.933 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:19.933062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:19.945 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:19.945796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:30.174 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:30.174206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:30.186 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:30.186447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:40.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:40.408950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:40.421 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:40.421711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:50.657 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:50.657455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:39:50.666 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:39:50.666866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:00.886 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:00.886562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:00.896 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:00.896936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:11.115 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:11.115359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:11.128 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:11.128512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:21.349 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:21.349214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:21.361 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:21.362368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:31.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:31.584020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:31.597 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:31.596743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:41.817 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:41.818187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:41.831 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:41.831433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:52.050 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:52.050329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:40:52.062 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:40:52.063189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:02.283 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:02.283966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:02.294 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:02.295226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:12.526 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:12.526196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:12.544 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:12.541912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:22.767 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:22.768112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:22.779 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:22.779674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:33.002 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:33.002005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:33.013 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:33.014207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:43.236 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:43.236506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:43.248 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:43.249087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:53.470 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:53.470743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:41:53.483 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:41:53.483623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:03.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:03.708635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:03.720 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:03.721219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:13.957 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:13.957908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:13.969 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:13.970084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:24.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:24.198365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:24.208 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:24.209095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:34.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:34.481739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:34.494 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:34.493882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:44.770 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:44.770541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:44.781 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:44.782507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:55.008 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:55.008682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:42:55.020 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:42:55.020848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:05.241 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:05.241677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:05.253 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:05.253911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:15.499 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:15.499669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:15.519 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:15.519744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:25.739 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:25.739884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:25.749 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:25.750227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:35.996 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:35.996374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:36.008 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:36.008619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:46.287 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:46.287039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:46.299 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:46.299700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:56.534 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:56.534681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:43:56.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:43:56.549193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:06.767 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:06.767699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:06.776 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:06.777289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:16.995 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:16.995420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:17.006 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:17.006723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:27.224 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:27.224980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:27.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:27.238329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:37.456 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:37.456623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:37.469 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:37.470208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:47.689 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:47.689470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:47.701 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:47.701859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:57.928 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:57.927926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:44:57.940 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:44:57.939992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:08.159 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:08.159827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:08.172 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:08.173045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:18.395 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:18.395081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:18.406 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:18.406941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:28.629 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:28.629344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:28.640 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:28.641401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:38.862 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:38.862497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:38.874 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:38.874869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:49.093 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:49.093876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:49.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:49.109670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:59.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:59.331262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:45:59.341 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:45:59.342142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:09.566 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:09.565748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:09.578 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:09.578642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:19.847 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:19.847934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:19.862 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:19.863118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:30.086 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:30.086491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:30.098 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:30.098936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:40.316 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:40.316354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:40.329 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:40.329043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:50.556 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:50.555967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:46:50.590 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:46:50.588084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:00.860 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:00.861123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:00.871 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:00.871875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:11.089 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:11.089242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:11.101 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:11.101471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:21.316 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:21.316867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:21.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:21.328719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:31.543 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:31.543429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:31.555 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:31.555280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:31.910 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:31.910394 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-08 23:47:31.899716359 +0000 UTC m=+3217.889081465 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:47:41.776 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:41.776055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:41.788 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:41.788971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:52.010 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:52.010196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:47:52.057 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:47:52.056857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:02.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:02.293949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:02.311 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:02.311977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:12.538 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:12.538739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:12.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:12.602486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:22.846 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:22.846197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:22.858 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:22.858867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:33.078 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:33.078876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:33.092 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:33.092821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:43.313 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:43.313548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:43.324 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:43.325121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:53.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:53.548315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:48:53.560 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:48:53.560431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:03.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:03.787250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:03.799 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:03.799401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:14.029 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:14.029078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:14.040 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:14.041225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:24.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:24.268238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:24.276 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:24.277301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:34.537 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:34.537476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:34.550 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:34.550372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:44.778 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:44.777989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:44.789 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:44.789923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:55.029 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:55.029578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:49:55.041 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:49:55.041687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:05.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:05.293487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:05.323 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:05.323713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:15.555 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:15.555571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:15.565 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:15.565141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:21.858 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:21.857953 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-08T23:50:22.510 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:22.510088 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:50:22.500+0000 7f2494102340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:50:22.511 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:50:22.500+0000 7f2494102340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:50:22.511 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:50:22.500+0000 7f2494102340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:50:22.511 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:50:22.511 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:22.510113 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:50:22.500+0000 7f2494102340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:50:22.512 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:50:22.500+0000 7f2494102340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:50:22.512 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08T23:50:22.500+0000 7f2494102340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:50:22.512 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:50:25.788 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:25.788821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:25.834 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:25.834109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:36.077 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:36.077171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:36.087 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:36.087634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:46.305 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:46.305785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:46.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:46.317796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:56.535 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:56.535345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:50:56.544 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:50:56.544984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:06.796 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:06.795907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:06.807 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:06.808261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:17.028 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:17.027369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:17.040 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:17.040223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:27.259 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:27.259092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:27.271 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:27.271441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:37.491 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:37.491463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:37.499 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:37.499848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:47.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:47.805395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:47.818 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:47.818264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:58.081 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:58.081300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:51:58.094 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:51:58.093073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:08.337 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:08.337056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:08.349 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:08.349250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:18.588 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:18.588592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:18.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:18.601583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:28.818 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:28.818234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:28.829 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:28.829675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:39.048 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:39.047873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:39.059 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:39.059310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:49.275 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:49.275019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:49.288 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:49.288565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:59.508 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:59.508160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:52:59.521 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:52:59.521561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:09.758 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:09.757666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:09.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:09.769184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:20.000 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:19.999899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:20.012 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:20.012159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:30.231 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:30.231439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:30.247 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:30.244524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:40.461 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:40.460674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:40.473 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:40.473280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:50.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:50.693029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:53:50.706 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:53:50.706524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:00.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:00.929561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:00.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:00.942479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:11.162 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:11.162184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:11.175 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:11.175574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:21.395 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:21.395398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:21.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:21.408939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:31.686 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:31.686494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:31.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:31.698594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:41.915 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:41.915229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:41.925 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:41.925760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:52.163 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:52.163406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:54:52.173 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:54:52.172964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:02.391 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:02.391370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:02.434 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:02.434523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:12.654 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:12.654218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:12.664 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:12.664382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:22.886 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:22.886265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:22.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:22.900159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:33.155 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:33.154351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:33.165 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:33.165542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:43.384 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:43.384477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:43.398 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:43.398121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:53.615 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:53.614455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:55:53.628 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:55:53.627977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:03.847 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:03.846688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:03.858 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:03.858138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:14.078 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:14.078077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:14.093 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:14.093455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:24.320 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:24.319848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:24.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:24.331656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:34.550 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:34.549758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:34.559 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:34.559182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:44.775 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:44.774631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:44.785 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:44.785313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:55.028 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:55.028420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:56:55.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:56:55.038636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:05.263 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:05.262760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:05.275 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:05.275046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:15.497 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:15.497214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:15.509 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:15.509152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:25.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:25.731781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:25.748 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:25.745692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:35.968 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:35.967626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:35.979 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:35.979473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:46.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:46.197809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:46.206 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:46.206244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:56.432 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:56.432168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:57:56.445 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:57:56.445803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:06.667 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:06.666980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:06.743 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:06.742551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:16.968 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:16.968010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:16.980 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:16.980576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:27.202 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:27.201168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:27.213 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:27.212923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:37.433 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:37.432893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:37.444 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:37.444384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:47.687 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:47.687354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:47.696 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:47.696691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:57.937 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:57.936954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:58:57.945 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:58:57.945795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:08.167 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:08.167170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:08.245 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:08.245626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:18.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:18.465262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:18.477 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:18.477321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:28.700 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:28.699417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:28.714 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:28.714490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:38.943 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:38.942423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:38.956 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:38.956212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:49.202 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:49.201608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:49.214 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:49.214225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:59.493 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:59.492591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-08T23:59:59.502 INFO:tasks.rook.operator.smithi176.stdout:2022-03-08 23:59:59.501998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:09.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:09.813473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:09.827 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:09.827021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:20.042 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:20.041362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:20.056 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:20.054695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:30.272 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:30.271756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:30.295 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:30.295643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:40.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:40.568323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:40.578 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:40.577455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:50.797 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:50.796152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:00:50.808 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:00:50.808666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:01.072 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:01.071748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:01.086 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:01.085246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:11.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:11.305801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:11.319 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:11.319032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:21.540 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:21.539873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:21.552 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:21.551960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:31.773 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:31.773160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:31.784 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:31.784737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:42.006 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:42.005219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:42.019 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:42.019616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:52.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:52.236622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:01:52.295 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:01:52.293995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:02.514 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:02.513612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:02.527 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:02.526667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:12.753 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:12.752201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:12.764 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:12.764524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:22.982 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:22.982036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:22.993 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:22.993477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:33.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:33.209188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:33.220 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:33.220363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:43.438 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:43.437550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:43.449 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:43.449045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:53.691 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:53.690875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:02:53.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:02:53.708412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:03.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:03.961155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:03.976 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:03.975446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:14.195 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:14.194686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:14.206 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:14.206426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:24.430 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:24.429914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:24.440 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:24.440372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:34.659 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:34.658661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:34.671 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:34.671623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:44.890 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:44.889971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:44.902 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:44.901836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:55.124 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:55.123432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:03:55.139 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:03:55.138394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:05.357 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:05.356763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:05.370 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:05.370198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:15.590 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:15.589512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:15.615 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:15.612200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:25.851 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:25.850275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:25.862 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:25.861512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:36.085 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:36.084911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:36.097 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:36.097157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:46.314 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:46.314030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:46.324 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:46.323803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:46.627 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:46.626242 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:04:46.617514598 +0000 UTC m=+4252.606879741 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:04:56.551 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:56.550373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:04:56.565 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:04:56.564709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:06.789 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:06.788946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:06.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:06.800590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:17.025 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:17.025050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:17.037 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:17.037232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:27.258 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:27.257577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:27.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:27.306376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:37.527 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:37.526360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:37.537 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:37.536595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:47.592 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:47.591855 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:05:47.549166893 +0000 UTC m=+4313.538532026 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:05:47.757 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:47.756579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:47.772 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:47.771650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:57.996 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:57.995598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:05:58.009 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:05:58.008316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:08.230 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:08.229594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:08.243 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:08.241747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:18.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:18.471123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:18.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:18.481291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:28.700 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:28.699765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:28.713 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:28.712973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:38.949 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:38.948683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:38.963 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:38.962838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:49.183 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:49.182811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:49.194 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:49.193690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:59.455 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:59.455134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:06:59.466 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:06:59.465795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:02.517 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:02.516861 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T00:07:03.159 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:03.157908 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:07:03.148+0000 7f470bef4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:07:03.159 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:07:03.148+0000 7f470bef4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:07:03.159 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:07:03.148+0000 7f470bef4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:07:03.160 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:07:03.160 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:03.157934 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:07:03.148+0000 7f470bef4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:07:03.160 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:07:03.148+0000 7f470bef4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:07:03.160 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:07:03.148+0000 7f470bef4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:07:03.161 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:07:09.681 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:09.680409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:09.719 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:09.700163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:19.917 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:19.916833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:19.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:19.928139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:30.145 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:30.145285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:30.160 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:30.157318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:40.377 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:40.376866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:40.388 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:40.387537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:50.608 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:50.607862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:07:50.622 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:07:50.621732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:00.855 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:00.854412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:00.864 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:00.864048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:11.079 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:11.078376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:11.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:11.110855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:21.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:21.329431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:21.342 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:21.341793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:31.659 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:31.658614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:31.668 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:31.667659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:41.898 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:41.897358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:41.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:41.908794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:52.138 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:52.137765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:08:52.153 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:08:52.153128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:02.379 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:02.378702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:02.392 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:02.391534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:12.616 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:12.616013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:12.629 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:12.628877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:22.848 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:22.848028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:22.858 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:22.858131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:33.076 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:33.075953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:33.088 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:33.087705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:43.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:43.305355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:43.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:43.317335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:53.535 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:53.534334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:09:53.547 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:09:53.547072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:03.772 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:03.771194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:03.782 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:03.782218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:14.012 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:14.010810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:14.019 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:14.019341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:24.239 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:24.238789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:24.248 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:24.248240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:34.479 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:34.477381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:34.489 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:34.488373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:44.737 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:44.736094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:44.763 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:44.762444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:55.036 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:55.035323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:10:55.046 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:10:55.044706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:05.259 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:05.258371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:05.271 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:05.271049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:15.490 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:15.489389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:15.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:15.504517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:25.746 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:25.745565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:25.759 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:25.759099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:35.987 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:35.980493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:35.993 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:35.993284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:46.242 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:46.241757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:46.255 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:46.254331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:56.473 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:56.472673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:11:56.485 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:11:56.483906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:06.707 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:06.706219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:06.717 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:06.716344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:16.936 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:16.935188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:16.949 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:16.948480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:27.170 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:27.169515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:27.181 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:27.181138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:37.402 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:37.401850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:37.414 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:37.414145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:47.635 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:47.634736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:47.647 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:47.646427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:57.869 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:57.868563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:12:57.877 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:12:57.876657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:08.129 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:08.128692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:08.140 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:08.140198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:18.358 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:18.357104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:18.371 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:18.370415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:28.592 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:28.591428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:28.602 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:28.601567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:38.862 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:38.861642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:38.910 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:38.909575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:49.131 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:49.129852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:49.143 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:49.143219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:59.363 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:59.361989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:13:59.378 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:13:59.377181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:09.595 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:09.594105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:09.607 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:09.606540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:19.833 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:19.832183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:19.843 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:19.842310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:30.064 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:30.063234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:30.076 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:30.076026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:40.295 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:40.294130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:40.308 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:40.307559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:50.539 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:50.538687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:14:50.553 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:14:50.553061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:00.804 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:00.802228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:00.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:00.813744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:11.044 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:11.043420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:11.092 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:11.092140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:21.341 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:21.340364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:21.351 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:21.350667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:31.573 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:31.572089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:31.586 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:31.585362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:41.800 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:41.800248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:41.811 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:41.811609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:52.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:52.031561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:15:52.043 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:15:52.043249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:02.266 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:02.265899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:02.277 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:02.277127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:12.499 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:12.498285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:12.509 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:12.508207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:22.755 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:22.755026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:22.771 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:22.770116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:32.990 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:32.989096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:33.002 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:33.002326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:43.224 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:43.223129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:43.236 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:43.236046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:53.453 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:53.452707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:16:53.463 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:16:53.461963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:03.685 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:03.684419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:03.696 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:03.695473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:13.915 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:13.914962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:13.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:13.928919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:24.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:24.414398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:24.423 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:24.422985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:34.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:34.651564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:34.664 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:34.663742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:44.885 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:44.884758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:44.896 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:44.896155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:55.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:55.118823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:17:55.131 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:17:55.131468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:05.358 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:05.357814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:05.371 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:05.370888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:15.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:15.619842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:15.631 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:15.631125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:25.877 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:25.876316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:25.888 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:25.887977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:36.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:36.107874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:36.126 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:36.126300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:46.355 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:46.354182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:46.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:46.367766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:56.614 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:56.613680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:18:56.625 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:18:56.625117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:06.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:06.843279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:06.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:06.899601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:17.145 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:17.145006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:17.157 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:17.157134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:27.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:27.414846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:27.427 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:27.426568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:37.649 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:37.648335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:37.659 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:37.659181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:47.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:47.874600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:47.889 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:47.888664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:58.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:58.108364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:19:58.121 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:19:58.121445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:08.344 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:08.343260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:08.356 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:08.356685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:18.575 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:18.575307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:18.584 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:18.583798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:28.807 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:28.806461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:28.832 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:28.831953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:39.089 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:39.088242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:39.106 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:39.106098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:49.329 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:49.328792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:49.342 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:49.341623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:59.636 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:59.635808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:20:59.644 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:20:59.644332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:09.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:09.899667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:09.910 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:09.910180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:20.134 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:20.133571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:20.147 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:20.146648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:30.374 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:30.373491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:30.386 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:30.385684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:40.640 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:40.639094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:40.656 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:40.656086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:50.982 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:50.981590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:21:50.995 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:21:50.995159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:01.239 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:01.238491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:01.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:01.249970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:01.552 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:01.552397 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:22:01.532656338 +0000 UTC m=+5287.522021494 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:22:11.470 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:11.469548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:11.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:11.481661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:21.697 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:21.696452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:21.730 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:21.729847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:31.977 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:31.977206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:31.986 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:31.985855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:42.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:42.208275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:42.229 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:42.229040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:52.447 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:52.446878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:22:52.458 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:22:52.457850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:02.681 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:02.681188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:02.695 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:02.694711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:12.911 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:12.910109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:12.923 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:12.923297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:23.144 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:23.144113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:23.155 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:23.155102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:33.373 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:33.372702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:33.385 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:33.384783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:43.166 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:43.166019 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T00:23:43.618 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:43.618360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:43.632 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:43.631379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:43.799 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:43.798579 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:23:43.790+0000 7f9c9de8f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:23:43.800 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:23:43.790+0000 7f9c9de8f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:23:43.800 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:23:43.790+0000 7f9c9de8f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:23:43.800 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:23:43.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:43.798607 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:23:43.790+0000 7f9c9de8f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:23:43.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:23:43.790+0000 7f9c9de8f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:23:43.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:23:43.790+0000 7f9c9de8f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:23:43.802 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:23:53.850 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:53.850059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:23:53.864 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:23:53.863721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:04.087 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:04.086954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:04.097 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:04.096615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:14.315 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:14.314971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:14.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:14.327366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:24.543 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:24.542169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:24.554 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:24.554502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:34.772 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:34.772070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:34.790 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:34.784080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:45.007 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:45.006409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:45.019 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:45.019098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:55.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:55.250201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:24:55.263 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:24:55.262069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:05.479 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:05.479159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:05.491 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:05.490922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:15.710 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:15.709835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:15.721 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:15.721590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:25.946 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:25.945273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:25.958 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:25.957826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:36.180 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:36.180125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:36.192 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:36.192003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:46.411 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:46.410741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:46.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:46.440127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:56.686 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:56.685524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:25:56.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:25:56.698259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:06.924 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:06.923458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:06.968 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:06.967792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:17.199 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:17.198913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:17.210 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:17.209764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:27.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:27.471029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:27.487 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:27.486885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:37.706 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:37.705497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:37.719 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:37.718102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:47.939 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:47.939016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:47.966 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:47.965952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:58.182 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:58.182110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:26:58.201 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:26:58.200895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:08.424 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:08.423682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:08.436 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:08.436413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:18.653 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:18.653277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:18.674 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:18.674074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:28.908 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:28.907471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:28.919 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:28.918849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:39.136 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:39.136246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:39.148 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:39.148496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:49.370 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:49.369928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:49.378 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:49.378408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:59.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:59.600183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:27:59.611 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:27:59.611437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:09.836 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:09.834918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:09.848 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:09.847825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:20.072 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:20.071609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:20.084 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:20.084237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:20.085 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:20.084262 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:30.307 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:30.307320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:30.318 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:30.318671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:40.540 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:40.539923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:40.553 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:40.552723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:50.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:50.768655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:28:50.779 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:28:50.778768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:01.004 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:01.003605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:01.016 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:01.016160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:11.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:11.236549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:11.248 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:11.248614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:21.470 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:21.469867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:21.485 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:21.484941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:31.710 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:31.710238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:31.721 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:31.721194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:41.976 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:41.976271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:41.988 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:41.988105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:52.207 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:52.207232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:29:52.223 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:29:52.222255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:02.447 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:02.446752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:02.459 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:02.459074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:12.694 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:12.693942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:12.706 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:12.706511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:22.963 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:22.962897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:22.975 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:22.974782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:33.224 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:33.223704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:33.233 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:33.233735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:43.463 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:43.463156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:43.477 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:43.477820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:53.715 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:53.714692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:30:53.726 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:30:53.726399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:03.945 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:03.944756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:03.956 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:03.956065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:14.176 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:14.175717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:14.214 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:14.214614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:24.441 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:24.440461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:24.456 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:24.454604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:34.673 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:34.672528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:34.686 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:34.686325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:44.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:44.908802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:44.919 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:44.919663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:55.153 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:55.152780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:31:55.166 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:31:55.166423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:05.390 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:05.389770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:05.404 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:05.404559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:15.626 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:15.626168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:15.638 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:15.637907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:25.863 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:25.863055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:25.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:25.874932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:36.104 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:36.103937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:36.117 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:36.117027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:46.339 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:46.338965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:46.353 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:46.352981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:56.577 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:56.576776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:32:56.589 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:32:56.589125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:06.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:06.813778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:06.825 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:06.825458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:17.043 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:17.043422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:17.058 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:17.058400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:27.280 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:27.280329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:27.295 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:27.294699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:37.521 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:37.521584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:37.536 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:37.535678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:47.754 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:47.754029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:47.767 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:47.767421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:57.983 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:57.983654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:33:57.997 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:33:57.997512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:08.221 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:08.220750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:08.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:08.250014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:18.468 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:18.467899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:18.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:18.548976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:28.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:28.768789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:28.782 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:28.781488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:39.095 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:39.094817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:39.107 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:39.107191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:49.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:49.328182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:49.375 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:49.375100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:59.720 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:59.720233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:34:59.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:34:59.731984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:09.978 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:09.978484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:09.993 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:09.993140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:20.212 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:20.212122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:20.225 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:20.225556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:30.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:30.443327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:30.456 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:30.455146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:40.676 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:40.675963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:40.684 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:40.684803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:50.930 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:50.929713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:35:50.940 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:35:50.940865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:01.170 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:01.169874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:01.181 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:01.181754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:11.403 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:11.402762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:11.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:11.415690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:21.634 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:21.634650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:21.645 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:21.645725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:31.867 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:31.867108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:31.879 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:31.879515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:42.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:42.099562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:42.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:42.113565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:52.337 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:52.337243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:36:52.392 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:36:52.389611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:02.612 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:02.611998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:02.626 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:02.626195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:12.845 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:12.845133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:12.857 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:12.857800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:23.080 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:23.079727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:23.095 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:23.095465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:33.311 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:33.310951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:33.321 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:33.321368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:43.545 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:43.545263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:43.556 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:43.556634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:53.778 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:53.778620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:37:53.823 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:37:53.820194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:04.049 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:04.049584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:04.062 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:04.062738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:14.282 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:14.282549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:14.302 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:14.295198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:24.530 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:24.529769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:24.542 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:24.542481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:34.783 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:34.783596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:34.798 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:34.797502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:45.017 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:45.017150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:45.031 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:45.031861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:55.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:55.250509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:38:55.261 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:38:55.261419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:05.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:05.480552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:05.496 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:05.496249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:15.744 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:15.744162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:15.762 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:15.762390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:25.979 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:25.978921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:25.991 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:25.991605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:36.226 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:36.226467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:36.238 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:36.238603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:46.456 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:46.456278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:46.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:46.471041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:56.714 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:56.714630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:39:56.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:39:56.724321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:06.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:06.941854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:06.953 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:06.954057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:17.219 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:17.219891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:17.229 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:17.229977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:17.362 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:17.363020 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:40:17.354010556 +0000 UTC m=+6383.343375732 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:40:23.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:23.805870 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T00:40:24.474 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:24.473563 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:40:24.464+0000 7f34cd60b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:40:24.474 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:40:24.464+0000 7f34cd60b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:40:24.475 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:40:24.464+0000 7f34cd60b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:40:24.475 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:40:24.475 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:24.473588 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:40:24.464+0000 7f34cd60b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:40:24.475 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:40:24.464+0000 7f34cd60b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:40:24.475 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:40:24.464+0000 7f34cd60b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:40:24.476 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:40:27.444 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:27.444508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:27.457 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:27.456141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:37.681 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:37.681300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:37.689 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:37.690015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:47.910 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:47.910463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:47.922 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:47.923049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:58.144 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:58.144324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:40:58.156 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:40:58.157152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:08.379 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:08.378909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:08.390 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:08.390750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:18.610 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:18.609870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:18.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:18.620845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:28.841 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:28.841556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:28.853 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:28.853178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:39.071 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:39.071231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:39.082 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:39.082773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:49.311 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:49.311063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:49.324 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:49.324046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:59.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:59.547585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:41:59.560 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:41:59.560674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:09.782 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:09.782607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:09.793 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:09.793891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:20.014 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:20.014854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:20.028 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:20.028665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:30.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:30.245711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:30.266 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:30.266710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:40.496 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:40.496113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:40.507 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:40.507166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:50.725 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:50.725732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:42:50.735 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:42:50.735149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:01.037 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:01.037011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:01.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:01.347374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:11.624 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:11.624147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:11.637 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:11.637299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:21.855 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:21.855334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:21.864 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:21.865050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:32.555 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:32.555810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:32.567 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:32.568127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:42.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:42.787340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:42.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:42.801312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:53.023 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:53.023685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:43:53.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:43:53.033201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:03.256 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:03.256116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:03.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:03.268746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:13.491 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:13.490951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:13.503 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:13.503748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:23.723 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:23.723071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:23.735 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:23.735742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:33.955 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:33.955017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:33.967 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:33.967108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:44.187 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:44.186712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:44.207 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:44.207128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:54.428 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:54.428365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:44:54.440 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:44:54.440362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:04.660 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:04.660154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:04.673 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:04.673210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:14.893 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:14.893324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:14.905 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:14.906206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:25.169 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:25.169131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:25.180 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:25.180859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:35.405 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:35.405301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:35.414 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:35.414947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:45.637 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:45.637368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:45.648 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:45.648851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:55.864 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:55.864572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:45:55.876 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:45:55.876336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:06.095 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:06.094733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:06.157 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:06.156964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:16.393 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:16.393271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:16.406 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:16.406394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:26.628 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:26.627659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:26.639 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:26.639586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:36.862 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:36.862490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:36.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:36.875318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:47.113 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:47.114151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:47.123 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:47.123679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:57.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:57.343388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:46:57.354 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:46:57.354542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:07.627 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:07.627574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:07.678 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:07.678612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:17.932 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:17.932969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:17.941 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:17.941573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:28.163 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:28.163274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:28.171 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:28.171449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:38.389 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:38.389363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:38.400 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:38.400692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:48.624 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:48.624525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:48.635 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:48.636173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:58.863 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:58.863090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:47:58.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:47:58.875080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:09.097 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:09.096908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:09.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:09.108462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:19.332 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:19.331894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:19.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:19.347097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:29.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:29.568300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:29.579 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:29.579843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:39.802 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:39.802103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:39.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:39.813752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:50.035 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:50.034715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:48:50.045 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:48:50.046274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:00.273 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:00.273400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:00.286 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:00.286207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:10.513 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:10.513759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:10.523 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:10.523905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:20.746 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:20.745939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:20.758 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:20.757973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:30.978 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:30.978522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:30.990 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:30.990371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:41.208 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:41.208880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:41.223 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:41.223532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:51.441 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:51.440936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:49:51.507 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:49:51.507765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:01.731 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:01.731038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:01.743 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:01.744088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:11.962 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:11.962918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:12.016 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:12.017242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:22.286 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:22.286451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:22.298 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:22.298966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:32.518 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:32.518344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:32.532 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:32.531718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:42.759 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:42.759326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:42.772 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:42.772449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:53.001 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:53.001156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:50:53.013 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:50:53.013335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:03.233 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:03.233392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:03.243 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:03.243476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:13.479 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:13.478683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:13.490 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:13.490344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:23.715 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:23.714632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:23.759 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:23.759647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:34.008 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:34.008844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:34.018 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:34.019104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:44.244 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:44.244420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:44.262 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:44.263159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:54.488 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:54.488664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:51:54.503 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:51:54.503428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:04.728 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:04.728899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:04.740 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:04.740576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:14.964 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:14.964735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:14.976 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:14.976476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:25.202 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:25.202283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:25.214 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:25.214812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:35.440 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:35.436588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:35.455 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:35.455960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:45.680 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:45.679893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:45.693 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:45.693601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:55.917 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:55.917791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:52:55.928 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:52:55.928902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:06.155 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:06.155430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:06.168 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:06.168958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:16.391 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:16.391921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:16.403 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:16.404373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:26.643 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:26.643413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:26.684 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:26.684796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:36.907 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:36.907796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:36.918 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:36.919215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:47.133 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:47.133602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:47.144 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:47.145000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:57.393 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:57.393199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:53:57.407 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:53:57.405935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:07.623 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:07.623977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:07.636 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:07.636867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:17.855 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:17.855800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:17.868 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:17.868904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:28.088 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:28.089062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:28.098 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:28.099403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:38.316 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:38.316967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:38.329 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:38.329720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:48.559 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:48.559708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:48.567 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:48.568201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:58.783 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:58.784078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:54:58.796 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:54:58.794860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:09.014 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:09.014386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:09.026 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:09.027540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:19.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:19.246439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:19.260 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:19.260463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:29.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:29.481603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:29.494 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:29.494414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:39.717 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:39.717454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:39.731 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:39.731227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:49.956 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:49.956148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:55:49.967 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:55:49.968412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:00.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:00.198244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:00.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:00.210279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:10.429 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:10.429659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:10.437 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:10.438067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:20.695 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:20.695230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:20.707 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:20.706819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:30.927 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:30.927870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:30.986 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:30.987215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:41.235 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:41.235653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:41.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:41.246156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:51.474 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:51.467514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:56:51.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:56:51.482247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:01.702 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:01.702938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:01.715 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:01.716326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:04.482 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:04.482770 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T00:57:05.075 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:05.074772 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:57:05.066+0000 7fcd29daf340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:57:05.075 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:57:05.066+0000 7fcd29daf340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:57:05.075 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:57:05.066+0000 7fcd29daf340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:57:05.076 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:57:05.076 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:05.074799 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:57:05.066+0000 7fcd29daf340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:57:05.076 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:57:05.066+0000 7fcd29daf340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:57:05.076 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T00:57:05.066+0000 7fcd29daf340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:57:05.077 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:57:11.935 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:11.935284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:11.948 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:11.948963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:22.189 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:22.189658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:22.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:22.199065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:32.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:32.416004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:32.427 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:32.427748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:32.502 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:32.502692 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 00:57:32.495263232 +0000 UTC m=+7418.484628385 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T00:57:42.663 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:42.662906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:42.702 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:42.702815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:52.917 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:52.917596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:57:52.928 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:57:52.929338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:03.152 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:03.153058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:03.164 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:03.165222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:13.417 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:13.418040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:13.429 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:13.428954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:23.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:23.650517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:23.662 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:23.662595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:33.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:33.876111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:33.883 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:33.884584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:44.102 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:44.102997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:44.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:44.112009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:54.333 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:54.333421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:58:54.341 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:58:54.342021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:04.561 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:04.561831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:04.575 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:04.575856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:14.823 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:14.823586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:14.835 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:14.835786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:25.057 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:25.057679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:25.068 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:25.069306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:35.287 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:35.287877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:35.299 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:35.298836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:45.544 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:45.544451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:45.555 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:45.555664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:55.778 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:55.778770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T00:59:55.789 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 00:59:55.790268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:06.022 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:06.022668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:06.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:06.054549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:16.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:16.302048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:16.309 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:16.310280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:26.526 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:26.526519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:26.537 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:26.538409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:36.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:36.788013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:36.799 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:36.799626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:47.019 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:47.019658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:47.031 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:47.031839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:57.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:57.250357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:00:57.259 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:00:57.259458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:07.480 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:07.481032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:07.491 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:07.491793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:17.711 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:17.711354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:17.723 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:17.724022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:27.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:27.942794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:27.954 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:27.954929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:38.172 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:38.173021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:38.185 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:38.185589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:48.403 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:48.403473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:48.416 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:48.417303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:58.641 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:58.641695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:01:58.651 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:01:58.652538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:08.881 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:08.881513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:08.893 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:08.893785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:19.152 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:19.152999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:19.163 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:19.164352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:29.402 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:29.403020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:29.417 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:29.417827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:39.638 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:39.638510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:39.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:39.650908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:49.870 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:49.870946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:02:49.931 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:02:49.932129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:00.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:00.330383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:00.391 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:00.392392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:10.630 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:10.630497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:10.643 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:10.644663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:20.865 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:20.865637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:20.873 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:20.874421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:31.093 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:31.093752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:31.105 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:31.106405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:41.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:41.328637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:41.340 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:41.341537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:51.561 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:51.561606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:03:51.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:03:51.569947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:01.794 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:01.794936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:01.803 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:01.804788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:12.024 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:12.024659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:12.034 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:12.035235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:22.256 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:22.257480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:22.267 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:22.268116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:32.494 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:32.494503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:32.506 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:32.507075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:42.725 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:42.726152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:42.737 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:42.737847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:52.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:52.962249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:04:52.974 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:04:52.974923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:03.196 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:03.196765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:03.208 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:03.209138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:13.430 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:13.430648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:13.442 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:13.443195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:23.664 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:23.664579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:23.676 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:23.676971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:33.899 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:33.900005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:33.912 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:33.913110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:44.131 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:44.131990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:44.141 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:44.142425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:54.375 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:54.376022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:05:54.385 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:05:54.384971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:04.600 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:04.601179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:04.612 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:04.613370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:14.833 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:14.834158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:14.847 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:14.847812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:25.067 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:25.068486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:25.079 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:25.080623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:35.302 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:35.302387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:35.312 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:35.313596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:45.542 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:45.542357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:45.555 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:45.556020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:55.775 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:55.776389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:06:55.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:06:55.788042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:06.005 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:06.006204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:06.020 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:06.020967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:16.243 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:16.243869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:16.255 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:16.256479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:26.480 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:26.480581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:26.493 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:26.494382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:36.719 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:36.719526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:36.730 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:36.730937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:46.951 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:46.952655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:46.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:46.962403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:57.186 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:57.184126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:07:57.192 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:07:57.193096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:07.405 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:07.405973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:07.418 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:07.419096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:17.636 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:17.636544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:17.649 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:17.649627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:27.868 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:27.869409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:27.881 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:27.882215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:38.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:38.115502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:38.126 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:38.127068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:48.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:48.348201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:48.359 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:48.360132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:58.581 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:58.581639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:08:58.591 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:08:58.592257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:08.818 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:08.818483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:08.831 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:08.831934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:19.083 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:19.083793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:19.095 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:19.095983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:29.319 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:29.319803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:29.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:29.331397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:39.558 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:39.555173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:39.569 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:39.570815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:49.819 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:49.820442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:09:49.839 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:09:49.839934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:00.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:00.054286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:00.064 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:00.065724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:10.315 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:10.315362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:10.326 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:10.327108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:20.569 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:20.570355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:20.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:20.585764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:30.803 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:30.804197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:30.815 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:30.816345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:41.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:41.038646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:41.049 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:41.050223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:51.269 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:51.270095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:10:51.280 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:10:51.281372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:01.515 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:01.515698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:01.528 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:01.528970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:11.751 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:11.751142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:11.760 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:11.761123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:21.981 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:21.981981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:21.993 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:21.994303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:32.208 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:32.209482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:32.222 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:32.222481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:42.438 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:42.438924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:42.450 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:42.451078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:52.672 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:52.672599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:11:52.683 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:11:52.684363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:02.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:02.930141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:02.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:02.942584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:13.161 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:13.161329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:13.175 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:13.175219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:23.404 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:23.404419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:23.418 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:23.419057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:33.639 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:33.639388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:33.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:33.651399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:43.870 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:43.871023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:43.882 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:43.882862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:54.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:54.108804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:12:54.120 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:12:54.121183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:04.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:04.343366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:04.354 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:04.353713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:14.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:14.568970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:14.578 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:14.579656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:24.806 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:24.807127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:24.818 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:24.819202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:35.040 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:35.040529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:35.053 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:35.054576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:45.133 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:45.134298 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T01:13:45.289 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:45.290490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:45.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:45.302309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:45.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:45.732401 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:13:45.724+0000 7fc31dd6f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:13:45.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:13:45.724+0000 7fc31dd6f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:13:45.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:13:45.724+0000 7fc31dd6f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:13:45.733 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:13:45.733 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:45.732424 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:13:45.724+0000 7fc31dd6f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:13:45.733 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:13:45.724+0000 7fc31dd6f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:13:45.733 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:13:45.724+0000 7fc31dd6f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:13:45.733 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:13:55.521 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:55.520689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:13:55.530 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:13:55.531815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:05.752 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:05.752879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:05.766 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:05.766728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:16.003 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:16.004139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:16.017 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:16.017768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:26.238 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:26.238586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:26.283 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:26.250334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:36.468 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:36.468797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:36.480 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:36.480939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:46.703 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:46.703286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:46.716 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:46.717459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:56.943 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:56.944422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:14:56.957 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:14:56.957911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:07.195 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:07.195739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:07.205 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:07.206318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:17.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:17.443463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:17.454 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:17.455075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:27.697 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:27.697962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:27.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:27.709614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:37.928 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:37.929040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:37.941 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:37.942077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:48.159 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:48.159883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:48.171 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:48.171555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:48.333 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:48.333325 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:15:48.325566615 +0000 UTC m=+8514.314931715 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:15:58.403 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:58.403654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:15:58.412 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:15:58.412797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:08.660 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:08.660775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:08.672 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:08.673472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:18.896 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:18.896745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:18.908 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:18.908794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:29.133 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:29.134080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:29.146 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:29.147245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:39.367 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:39.367464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:39.379 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:39.380202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:49.598 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:49.598646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:49.611 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:49.610925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:59.831 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:59.832164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:16:59.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:16:59.845275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:10.061 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:10.061904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:10.073 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:10.074285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:20.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:20.297136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:20.312 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:20.313211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:30.533 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:30.534198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:30.546 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:30.547169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:40.769 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:40.769343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:40.781 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:40.781816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:51.063 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:51.063922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:17:51.079 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:17:51.078921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:01.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:01.342398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:01.385 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:01.386105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:11.621 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:11.621132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:11.636 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:11.637484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:21.860 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:21.861027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:21.872 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:21.872781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:32.092 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:32.092756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:32.106 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:32.107317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:42.327 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:42.327628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:42.338 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:42.338902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:52.558 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:52.558632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:18:52.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:18:52.568941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:02.786 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:02.787131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:02.857 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:02.857795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:13.093 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:13.093833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:13.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:13.110732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:23.327 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:23.327164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:23.340 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:23.340609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:33.560 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:33.560398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:33.572 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:33.572279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:43.795 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:43.795561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:43.809 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:43.809752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:54.089 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:54.089258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:19:54.100 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:19:54.100665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:04.322 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:04.320796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:04.333 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:04.334151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:14.551 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:14.551559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:14.562 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:14.562970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:24.783 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:24.783405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:24.796 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:24.796120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:35.017 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:35.017424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:35.027 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:35.028603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:45.249 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:45.249311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:45.264 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:45.264764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:55.481 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:55.481815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:20:55.494 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:20:55.494576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:05.714 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:05.714400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:05.726 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:05.726990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:15.950 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:15.950535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:15.962 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:15.963421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:26.186 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:26.186464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:26.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:26.198281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:36.425 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:36.425351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:36.432 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:36.433173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:46.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:46.652863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:46.663 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:46.664373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:56.883 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:56.883512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:21:56.895 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:21:56.894577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:07.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:07.111934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:07.122 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:07.123408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:17.381 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:17.381245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:17.393 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:17.393463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:27.639 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:27.639689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:27.645 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:27.646812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:38.006 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:38.005762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:38.019 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:38.019824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:48.289 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:48.289957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:48.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:48.302370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:58.597 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:58.597871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:22:58.607 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:22:58.607789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:08.871 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:08.871634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:08.880 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:08.881555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:19.148 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:19.148517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:19.158 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:19.158957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:29.393 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:29.393357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:29.402 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:29.402861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:39.622 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:39.622575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:39.635 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:39.635637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:49.871 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:49.871815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:23:49.881 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:23:49.881801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:00.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:00.099904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:00.140 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:00.140901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:10.361 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:10.362025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:10.430 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:10.430575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:20.733 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:20.734091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:20.746 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:20.744764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:30.972 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:30.973272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:30.983 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:30.984222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:41.208 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:41.208733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:41.221 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:41.221787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:51.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:51.627178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:24:51.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:24:51.639722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:01.870 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:01.870720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:01.883 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:01.883689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:12.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:12.108654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:12.120 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:12.121026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:22.449 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:22.449746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:22.460 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:22.461632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:32.738 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:32.738668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:32.843 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:32.843803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:43.060 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:43.060638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:43.074 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:43.074814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:53.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:53.294352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:25:53.307 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:25:53.308004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:03.530 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:03.526628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:03.539 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:03.539979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:13.808 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:13.808685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:13.820 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:13.821075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:24.049 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:24.049604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:24.059 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:24.060306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:34.277 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:34.277977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:34.291 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:34.292217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:44.507 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:44.507573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:44.519 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:44.520234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:54.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:54.853063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:26:54.865 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:26:54.866068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:05.090 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:05.090959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:05.101 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:05.102473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:15.316 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:15.317203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:15.326 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:15.327240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:25.611 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:25.611410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:25.623 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:25.623957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:35.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:35.844435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:35.857 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:35.858208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:46.076 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:46.076363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:46.087 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:46.088057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:56.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:56.305690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:27:56.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:27:56.317905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:06.541 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:06.535874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:06.570 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:06.571326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:16.796 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:16.796639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:16.810 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:16.809587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:27.025 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:27.026436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:27.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:27.038741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:27.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:27.038773 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:37.257 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:37.257752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:37.274 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:37.274263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:47.498 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:47.498464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:47.508 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:47.509595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:57.731 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:57.732240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:28:57.744 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:28:57.744963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:07.987 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:07.987662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:08.003 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:08.004149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:18.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:18.246659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:18.258 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:18.259043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:28.476 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:28.477392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:28.487 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:28.488197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:38.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:38.732756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:38.744 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:38.744921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:48.989 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:48.989392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:49.001 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:49.002209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:59.220 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:59.220564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:29:59.283 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:29:59.284405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:09.503 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:09.503346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:09.514 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:09.515423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:19.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:19.732576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:19.750 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:19.750641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:25.740 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:25.740467 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T01:30:26.361 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:26.361435 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:30:26.353+0000 7f9ed03fd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:30:26.362 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:30:26.353+0000 7f9ed03fd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:30:26.362 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:30:26.353+0000 7f9ed03fd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:30:26.362 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:30:26.362 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:26.361462 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:30:26.353+0000 7f9ed03fd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:30:26.363 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:30:26.353+0000 7f9ed03fd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:30:26.363 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:30:26.353+0000 7f9ed03fd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:30:26.363 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:30:29.968 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:29.968931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:29.981 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:29.982079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:40.196 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:40.197038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:40.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:40.209905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:50.460 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:50.460396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:30:50.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:30:50.472185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:00.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:00.691462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:00.704 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:00.703781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:10.926 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:10.926920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:10.938 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:10.939177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:21.164 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:21.165213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:21.174 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:21.174969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:31.395 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:31.396042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:31.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:31.410402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:41.632 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:41.632742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:41.643 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:41.644242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:51.866 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:51.867205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:31:51.914 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:31:51.915125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:02.135 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:02.136197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:02.146 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:02.147822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:02.399 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:02.399798 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:32:02.39246802 +0000 UTC m=+9488.381833174 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:32:12.368 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:12.368987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:12.375 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:12.376694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:22.591 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:22.592617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:22.603 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:22.603764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:32.829 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:32.829773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:32.840 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:32.841424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:43.068 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:43.068984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:43.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:43.114662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:53.332 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:53.332701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:32:53.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:32:53.344091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:03.590 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:03.590680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:03.599 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:03.600366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:13.820 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:13.820531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:13.830 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:13.830825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:24.045 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:24.045902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:24.056 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:24.056822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:34.318 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:34.318812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:34.326 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:34.326723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:44.543 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:44.544141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:44.563 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:44.556601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:54.779 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:54.780121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:33:54.791 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:33:54.792578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:05.012 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:05.013102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:05.025 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:05.026046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:15.247 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:15.247798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:15.261 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:15.261004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:25.480 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:25.480164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:25.491 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:25.491702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:35.712 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:35.712261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:35.723 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:35.723721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:45.945 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:45.946070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:45.957 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:45.957686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:56.176 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:56.177014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:34:56.187 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:34:56.188768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:06.428 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:06.428948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:06.439 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:06.439920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:16.660 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:16.660556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:16.673 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:16.673633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:26.892 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:26.893150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:26.906 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:26.906023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:37.122 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:37.123093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:37.134 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:37.134524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:47.356 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:47.357270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:47.367 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:47.366581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:57.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:57.586138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:35:57.596 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:35:57.597073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:07.821 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:07.821570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:07.828 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:07.829784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:18.090 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:18.090599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:18.101 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:18.102688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:28.320 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:28.320811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:28.356 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:28.356490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:38.579 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:38.579970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:38.590 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:38.591275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:48.815 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:48.815862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:48.827 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:48.827681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:59.050 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:59.050998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:36:59.063 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:36:59.064078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:09.286 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:09.286814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:09.298 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:09.298707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:19.589 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:19.590011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:19.601 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:19.601795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:29.815 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:29.815525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:29.827 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:29.827659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:40.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:40.115118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:40.123 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:40.123959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:50.341 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:50.341765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:37:50.356 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:37:50.357288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:00.576 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:00.576340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:00.592 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:00.590713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:10.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:10.814558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:10.823 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:10.824184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:21.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:21.038662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:21.117 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:21.117381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:31.360 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:31.360284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:31.371 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:31.371961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:41.596 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:41.596677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:41.608 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:41.609192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:51.824 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:51.824633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:38:51.831 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:38:51.832297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:02.056 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:02.055973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:02.066 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:02.067460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:12.286 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:12.286715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:12.295 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:12.295546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:22.527 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:22.527708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:22.537 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:22.537951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:32.755 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:32.755824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:32.767 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:32.768072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:42.988 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:42.988823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:43.000 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:43.001182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:53.220 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:53.220454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:39:53.234 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:39:53.234789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:03.458 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:03.458752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:03.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:03.471952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:13.694 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:13.695016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:13.706 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:13.706950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:23.930 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:23.930918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:23.944 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:23.944802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:34.168 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:34.168731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:34.181 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:34.181425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:44.404 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:44.404346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:44.417 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:44.418230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:54.644 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:54.644504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:40:54.657 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:40:54.658138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:04.882 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:04.882635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:04.895 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:04.895785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:15.123 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:15.123472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:15.140 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:15.140674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:25.367 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:25.368241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:25.379 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:25.380190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:35.598 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:35.598544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:35.611 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:35.611199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:45.828 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:45.828985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:45.840 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:45.841188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:56.059 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:56.059776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:41:56.072 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:41:56.072759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:06.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:06.294019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:06.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:06.306910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:16.532 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:16.532858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:16.542 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:16.542812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:26.763 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:26.763708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:26.779 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:26.778640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:36.996 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:36.996803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:37.007 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:37.008183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:47.232 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:47.233115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:47.245 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:47.246451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:57.468 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:57.468437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:42:57.482 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:42:57.481130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:07.697 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:07.698211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:07.708 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:07.709017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:17.925 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:17.925241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:17.937 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:17.938097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:28.160 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:28.160666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:28.171 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:28.171128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:38.392 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:38.392196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:38.405 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:38.405721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:48.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:48.633677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:48.645 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:48.645603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:58.880 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:58.880573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:43:58.919 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:43:58.920192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:09.142 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:09.142448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:09.151 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:09.151683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:19.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:19.409799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:19.420 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:19.421414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:29.643 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:29.643616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:29.653 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:29.654630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:39.876 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:39.876190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:39.888 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:39.888815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:50.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:50.108836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:44:50.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:44:50.119756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:00.349 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:00.349110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:00.362 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:00.363265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:10.587 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:10.587465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:10.600 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:10.601147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:20.816 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:20.817086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:20.829 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:20.828688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:31.045 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:31.046007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:31.094 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:31.094210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:41.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:41.331827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:41.343 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:41.343871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:51.572 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:51.572302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:45:51.590 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:45:51.591076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:01.804 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:01.805289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:01.816 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:01.817223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:12.069 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:12.069088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:12.080 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:12.081173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:22.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:22.306822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:22.318 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:22.318878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:32.537 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:32.537968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:32.549 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:32.550055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:42.768 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:42.768713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:42.781 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:42.782430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:53.002 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:53.002955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:46:53.014 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:46:53.014834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:03.234 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:03.234417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:03.247 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:03.247610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:06.377 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:06.378431 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T01:47:07.053 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:07.052839 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:47:07.046+0000 7f3be61c8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:47:07.053 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:47:07.046+0000 7f3be61c8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:47:07.053 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:47:07.046+0000 7f3be61c8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:47:07.053 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:47:07.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:07.052865 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:47:07.046+0000 7f3be61c8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:47:07.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:47:07.046+0000 7f3be61c8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:47:07.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T01:47:07.046+0000 7f3be61c8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:47:07.054 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:47:13.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:13.471673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:13.483 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:13.483677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:23.705 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:23.705076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:23.718 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:23.719532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:33.943 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:33.943098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:33.952 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:33.952981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:44.175 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:44.175415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:44.188 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:44.189377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:54.414 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:54.414932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:47:54.427 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:47:54.427995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:04.655 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:04.653908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:04.666 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:04.666870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:14.892 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:14.893146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:14.904 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:14.904990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:25.143 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:25.143259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:25.157 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:25.157884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:35.382 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:35.382548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:35.393 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:35.394304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:45.617 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:45.617338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:45.628 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:45.628294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:55.884 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:55.885278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:48:55.896 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:48:55.897543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:06.116 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:06.116453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:06.128 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:06.128293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:16.363 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:16.363864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:16.375 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:16.375663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:26.589 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:26.590025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:26.629 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:26.630261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:36.846 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:36.846526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:36.857 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:36.858068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:47.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:47.100301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:47.116 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:47.116648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:57.334 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:57.334902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:49:57.344 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:49:57.345409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:07.558 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:07.558952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:07.623 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:07.624314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:17.865 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:17.865763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:17.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:17.875392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:18.201 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:18.202137 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:50:18.191032437 +0000 UTC m=+10584.180397578 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:50:28.093 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:28.093356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:28.105 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:28.106469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:38.325 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:38.326125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:38.337 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:38.338176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:48.564 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:48.564760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:48.576 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:48.576786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:58.795 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:58.795015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:50:58.808 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:50:58.809313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:09.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:09.032417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:09.043 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:09.044465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:19.227 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:19.228007 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 01:51:19.222102176 +0000 UTC m=+10645.211467272 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T01:51:19.263 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:19.264353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:19.277 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:19.277967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:29.501 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:29.500999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:29.512 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:29.513028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:39.733 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:39.733614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:39.746 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:39.746922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:49.968 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:49.968177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:51:50.020 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:51:50.021099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:00.240 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:00.240865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:00.250 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:00.250585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:10.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:10.472071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:10.483 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:10.483690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:20.704 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:20.704551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:20.716 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:20.716701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:30.941 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:30.941381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:30.955 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:30.955897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:41.174 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:41.174312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:41.184 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:41.185069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:51.406 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:51.406713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:52:51.440 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:52:51.429537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:01.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:01.650594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:01.663 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:01.664002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:11.958 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:11.958469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:11.970 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:11.970696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:22.222 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:22.223154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:22.241 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:22.242360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:32.479 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:32.479877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:32.558 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:32.558684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:42.811 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:42.811444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:42.818 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:42.819666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:53.059 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:53.059684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:53:53.071 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:53:53.071545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:03.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:03.296749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:03.313 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:03.313665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:13.545 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:13.546027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:13.557 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:13.558435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:23.778 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:23.778469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:23.788 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:23.789637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:34.007 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:34.008224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:34.016 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:34.017598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:44.241 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:44.241100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:44.255 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:44.255915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:54.475 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:54.475748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:54:54.487 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:54:54.487708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:04.707 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:04.707330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:04.716 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:04.717287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:14.941 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:14.941399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:14.954 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:14.954961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:25.173 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:25.173675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:25.186 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:25.186475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:35.404 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:35.404640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:35.412 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:35.413283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:45.633 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:45.634141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:45.646 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:45.646991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:55.888 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:55.889326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:55:55.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:55:55.901022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:06.124 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:06.124378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:06.134 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:06.135095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:16.362 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:16.362641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:16.389 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:16.389799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:26.640 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:26.641108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:26.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:26.651962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:36.874 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:36.875332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:36.885 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:36.885635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:47.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:47.112009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:47.124 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:47.123427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:57.344 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:57.344321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:56:57.360 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:56:57.357526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:07.587 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:07.588283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:07.598 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:07.598767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:17.851 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:17.851324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:17.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:17.876438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:28.115 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:28.116195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:28.126 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:28.127456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:38.370 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:38.370277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:38.383 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:38.384160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:48.606 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:48.606345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:48.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:48.621655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:58.918 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:58.918287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:57:58.929 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:57:58.929358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:09.146 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:09.146526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:09.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:09.198295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:19.571 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:19.571571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:19.582 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:19.582689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:29.908 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:29.908693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:29.920 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:29.920771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:40.169 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:40.169516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:40.181 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:40.182301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:50.533 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:50.533437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:58:50.545 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:58:50.546277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:00.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:00.805557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:00.815 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:00.815890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:11.040 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:11.040918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:11.049 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:11.049683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:21.266 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:21.266242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:21.279 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:21.280274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:31.500 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:31.500530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:31.512 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:31.512729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:41.733 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:41.733447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:41.747 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:41.747663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:51.967 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:51.967479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T01:59:51.977 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 01:59:51.977987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:02.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:02.198911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:02.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:02.210561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:12.435 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:12.436488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:12.446 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:12.447300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:22.669 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:22.670267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:22.682 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:22.683189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:32.910 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:32.911074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:32.922 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:32.922978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:43.142 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:43.142419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:43.189 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:43.189557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:53.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:53.409880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:00:53.423 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:00:53.423962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:03.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:03.642603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:03.655 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:03.656267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:13.876 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:13.876286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:13.887 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:13.887849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:24.107 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:24.108170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:24.118 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:24.118731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:34.338 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:34.338562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:34.358 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:34.358261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:44.585 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:44.586042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:44.599 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:44.600413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:54.859 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:54.860146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:01:54.868 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:01:54.868800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:05.087 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:05.087890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:05.099 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:05.100401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:15.318 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:15.318987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:15.328 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:15.329608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:25.549 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:25.550091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:25.562 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:25.562751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:35.788 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:35.788461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:35.799 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:35.800148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:46.020 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:46.020589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:46.031 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:46.032145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:56.255 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:56.256394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:02:56.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:02:56.268311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:06.507 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:06.508060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:06.521 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:06.521701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:16.740 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:16.740436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:16.753 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:16.754029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:26.973 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:26.973784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:26.982 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:26.983289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:37.202 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:37.203357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:37.215 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:37.216063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:47.064 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:47.064866 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T02:03:47.467 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:47.468338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:47.485 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:47.486003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:47.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:47.724344 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:03:47.716+0000 7f7eafea9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:03:47.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:03:47.716+0000 7f7eafea9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:03:47.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:03:47.716+0000 7f7eafea9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:03:47.725 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:03:47.725 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:47.724364 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:03:47.716+0000 7f7eafea9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:03:47.725 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:03:47.716+0000 7f7eafea9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:03:47.725 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:03:47.716+0000 7f7eafea9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:03:47.726 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:03:57.711 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:57.711252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:03:57.722 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:03:57.722958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:07.945 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:07.946035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:07.958 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:07.958580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:18.173 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:18.174076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:18.185 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:18.186628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:28.408 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:28.408216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:28.419 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:28.419903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:38.647 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:38.641969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:38.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:38.653662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:48.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:48.875452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:48.894 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:48.895161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:59.131 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:59.131589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:04:59.144 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:04:59.145077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:09.363 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:09.363029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:09.411 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:09.412257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:19.626 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:19.626501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:19.663 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:19.663589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:29.885 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:29.886238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:29.897 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:29.898285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:40.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:40.119573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:40.130 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:40.131125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:50.348 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:50.348692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:05:50.438 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:05:50.438197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:00.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:00.652741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:00.659 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:00.660010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:10.877 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:10.877448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:10.888 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:10.889007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:21.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:21.110524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:21.123 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:21.123560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:31.355 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:31.355221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:31.369 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:31.368186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:41.589 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:41.589665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:41.603 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:41.603701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:51.827 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:51.827896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:06:51.840 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:06:51.840849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:02.064 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:02.064471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:02.072 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:02.072878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:12.296 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:12.297031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:12.308 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:12.308870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:22.527 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:22.527590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:22.539 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:22.540334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:32.782 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:32.783050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:32.795 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:32.796186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:43.016 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:43.016705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:43.041 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:43.041902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:53.265 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:53.265281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:07:53.277 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:07:53.277608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:03.510 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:03.510947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:03.523 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:03.523919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:13.743 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:13.743332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:13.756 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:13.756348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:23.976 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:23.976557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:23.988 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:23.988470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:34.208 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:34.208157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:34.223 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:34.222601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:34.248 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:34.248820 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:08:34.227356255 +0000 UTC m=+11680.216721348 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:08:44.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:44.443666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:44.459 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:44.458863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:54.673 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:54.673640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:08:54.685 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:08:54.686489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:04.904 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:04.904870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:04.917 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:04.917401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:15.135 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:15.135840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:15.146 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:15.147169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:25.363 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:25.363695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:25.377 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:25.377531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:35.595 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:35.595710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:35.607 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:35.608135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:45.828 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:45.828733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:45.840 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:45.840926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:56.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:56.110659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:09:56.123 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:09:56.124165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:06.340 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:06.340849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:06.354 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:06.354530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:16.571 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:16.572233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:16.582 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:16.583283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:26.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:26.801699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:26.814 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:26.814074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:37.034 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:37.034101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:37.047 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:37.047924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:47.269 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:47.269286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:47.283 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:47.282797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:57.506 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:57.501865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:10:57.514 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:10:57.514407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:07.736 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:07.737248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:07.747 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:07.747893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:17.968 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:17.968701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:17.984 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:17.984119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:28.210 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:28.210615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:28.225 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:28.224278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:38.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:38.443686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:38.455 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:38.456152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:48.673 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:48.673291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:48.685 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:48.685982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:58.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:58.909654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:11:58.920 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:11:58.920972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:09.187 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:09.187744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:09.198 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:09.199000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:19.437 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:19.437713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:19.464 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:19.464630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:29.685 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:29.685381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:29.697 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:29.698081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:39.974 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:39.974086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:39.987 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:39.987458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:50.228 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:50.228872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:12:50.279 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:12:50.278655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:00.520 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:00.520300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:00.529 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:00.530027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:10.760 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:10.760532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:10.773 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:10.772377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:20.992 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:20.992093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:21.005 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:21.006483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:31.273 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:31.273513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:31.283 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:31.283606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:41.500 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:41.500350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:41.512 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:41.512766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:51.753 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:51.753850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:13:51.765 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:13:51.766159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:01.985 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:01.985139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:01.997 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:01.998006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:12.213 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:12.213160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:12.223 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:12.224179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:22.437 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:22.437640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:22.449 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:22.449615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:32.668 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:32.668438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:32.683 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:32.683320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:42.901 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:42.901776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:42.912 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:42.912433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:53.137 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:53.137129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:14:53.149 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:14:53.149885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:03.399 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:03.398814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:03.412 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:03.412879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:13.630 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:13.630537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:13.643 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:13.643984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:23.862 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:23.862283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:23.874 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:23.874664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:34.096 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:34.096701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:34.109 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:34.109265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:44.326 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:44.326230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:44.340 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:44.340879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:54.557 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:54.556856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:15:54.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:15:54.568368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:04.785 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:04.785961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:04.799 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:04.799934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:15.037 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:15.037006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:15.048 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:15.049111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:25.269 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:25.269079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:25.283 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:25.283804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:35.503 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:35.503153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:35.514 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:35.514467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:45.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:45.801164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:45.812 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:45.812719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:56.035 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:56.035459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:16:56.048 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:16:56.047808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:06.262 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:06.262726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:06.275 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:06.275632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:16.509 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:16.509510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:16.522 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:16.522988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:26.741 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:26.741425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:26.755 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:26.755172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:36.991 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:36.991841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:37.003 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:37.003802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:47.224 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:47.225179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:47.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:47.237800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:57.455 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:57.454931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:17:57.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:17:57.465957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:07.684 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:07.684064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:07.696 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:07.696746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:17.917 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:17.917676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:17.961 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:17.962045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:28.180 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:28.180282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:28.193 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:28.194278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:38.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:38.415895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:38.427 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:38.428132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:48.654 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:48.654543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:48.664 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:48.664920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:58.888 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:58.888898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:18:58.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:18:58.900813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:09.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:09.119232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:09.131 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:09.131184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:19.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:19.346815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:19.359 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:19.359378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:29.576 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:29.576449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:29.588 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:29.588920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:39.805 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:39.805080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:39.813 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:39.813649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:50.037 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:50.037519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:19:50.049 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:19:50.049565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:00.270 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:00.270158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:00.281 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:00.282354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:10.500 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:10.500186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:10.514 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:10.514157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:20.739 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:20.733638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:20.746 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:20.746313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:27.781 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:27.781859 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T02:20:28.451 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:28.450718 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:20:28.441+0000 7efd305d7340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:20:28.451 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:20:28.441+0000 7efd305d7340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:20:28.451 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:20:28.441+0000 7efd305d7340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:20:28.452 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:20:28.452 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:28.450744 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:20:28.441+0000 7efd305d7340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:20:28.452 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:20:28.441+0000 7efd305d7340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:20:28.452 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:20:28.441+0000 7efd305d7340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:20:28.453 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:20:30.963 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:30.963627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:30.975 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:30.975735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:41.207 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:41.207206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:41.251 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:41.233770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:51.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:51.505586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:20:51.515 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:20:51.516066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:01.737 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:01.737214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:01.751 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:01.751352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:11.972 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:11.972622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:11.983 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:11.984110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:22.221 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:22.221231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:22.235 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:22.236143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:32.515 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:32.515680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:32.524 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:32.524779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:42.789 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:42.789577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:42.803 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:42.803310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:53.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:53.032756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:21:53.048 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:21:53.048325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:03.268 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:03.268060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:03.279 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:03.279225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:13.690 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:13.690692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:13.700 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:13.700529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:23.976 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:23.976702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:23.989 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:23.990063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:34.210 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:34.209834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:34.222 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:34.223134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:44.576 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:44.576346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:44.590 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:44.591054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:54.810 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:54.809975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:22:54.822 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:22:54.822769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:05.038 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:05.038257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:05.051 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:05.051886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:15.453 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:15.453044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:15.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:15.465533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:25.737 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:25.737323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:25.787 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:25.787356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:36.041 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:36.040896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:36.053 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:36.053679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:46.288 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:46.288948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:46.301 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:46.301548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:56.537 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:56.537507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:23:56.553 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:23:56.551984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:06.800 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:06.800167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:06.811 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:06.811605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:17.308 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:17.307992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:17.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:17.318015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:27.548 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:27.548603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:27.557 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:27.558259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:37.823 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:37.823552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:37.875 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:37.875246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:48.388 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:48.388660 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:24:48.22491647 +0000 UTC m=+12654.214281678 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:24:48.460 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:48.460904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:48.471 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:48.471884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:58.688 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:58.688671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:24:58.700 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:24:58.700743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:08.925 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:08.925880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:08.938 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:08.938452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:19.307 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:19.307352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:19.508 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:19.508677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:29.728 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:29.728671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:29.743 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:29.743631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:39.963 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:39.963004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:39.977 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:39.976427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:50.196 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:50.196411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:25:50.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:25:50.209502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:00.430 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:00.430845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:00.443 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:00.444195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:10.665 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:10.665904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:10.676 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:10.677004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:20.899 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:20.899321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:20.909 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:20.910060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:31.131 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:31.131589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:31.143 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:31.143772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:41.385 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:41.385257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:41.399 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:41.399331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:51.650 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:51.650208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:26:51.661 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:26:51.661625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:01.881 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:01.881367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:01.893 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:01.893899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:12.115 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:12.115197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:12.128 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:12.128787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:22.347 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:22.346989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:22.358 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:22.358849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:32.576 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:32.576618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:32.586 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:32.586469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:42.807 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:42.807167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:42.819 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:42.819954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:53.044 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:53.044499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:27:53.056 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:27:53.056849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:03.276 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:03.276423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:03.288 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:03.288439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:13.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:13.505717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:13.517 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:13.517650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:23.741 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:23.741231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:23.790 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:23.790944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:34.013 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:34.012829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:34.025 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:34.024974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:34.026 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:34.025012 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:44.248 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:44.248064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:44.258 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:44.258907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:54.528 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:54.528923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:28:54.543 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:28:54.542153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:04.758 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:04.758978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:04.767 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:04.767990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:15.001 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:15.001486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:15.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:15.032535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:25.257 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:25.257863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:25.266 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:25.267068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:35.486 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:35.486240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:35.498 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:35.498317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:45.716 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:45.716714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:45.728 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:45.728439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:55.947 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:55.946847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:29:55.962 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:29:55.963203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:06.239 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:06.239372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:06.253 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:06.253672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:16.476 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:16.476205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:16.483 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:16.483978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:26.702 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:26.702582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:26.716 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:26.717089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:36.942 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:36.942196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:36.952 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:36.953089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:47.170 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:47.170925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:47.180 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:47.181332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:57.399 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:57.398804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:30:57.410 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:30:57.410970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:07.629 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:07.629039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:07.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:07.643037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:17.897 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:17.896804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:17.924 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:17.924357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:28.146 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:28.146528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:28.159 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:28.159884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:38.403 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:38.402764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:38.415 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:38.415522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:48.634 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:48.633759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:48.656 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:48.656113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:58.880 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:58.879877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:31:58.890 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:31:58.890784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:09.111 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:09.110951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:09.123 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:09.123405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:19.423 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:19.422972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:19.434 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:19.434184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:29.655 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:29.655309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:29.874 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:29.668311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:39.889 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:39.888927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:39.900 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:39.900877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:50.148 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:50.148481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:32:50.157 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:32:50.157778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:00.387 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:00.387509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:00.426 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:00.401773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:10.620 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:10.620536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:10.635 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:10.632665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:20.849 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:20.849029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:20.861 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:20.861804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:31.078 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:31.078573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:31.090 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:31.090109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:41.383 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:41.383421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:41.395 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:41.395285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:51.639 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:51.639210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:33:51.652 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:33:51.652521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:01.874 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:01.874362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:01.886 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:01.886324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:12.108 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:12.107786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:12.119 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:12.119727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:22.340 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:22.339653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:22.352 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:22.351555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:32.569 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:32.569277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:32.579 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:32.579589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:42.797 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:42.797616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:42.815 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:42.815779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:53.041 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:53.041090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:34:53.054 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:34:53.054426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:03.274 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:03.274186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:03.288 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:03.288244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:13.510 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:13.509686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:13.522 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:13.522692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:23.747 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:23.745796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:23.758 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:23.757674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:33.981 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:33.980767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:33.994 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:33.994683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:44.217 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:44.217464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:44.230 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:44.230085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:54.455 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:54.454996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:35:54.469 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:35:54.468938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:04.742 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:04.742502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:04.757 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:04.757378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:15.064 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:15.064385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:15.073 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:15.073651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:25.294 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:25.294036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:25.306 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:25.306086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:35.530 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:35.529949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:35.542 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:35.542356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:45.763 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:45.763051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:45.774 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:45.774670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:55.994 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:55.993596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:36:56.052 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:36:56.052134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:06.273 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:06.272508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:06.286 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:06.286145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:08.463 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:08.462625 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T02:37:09.082 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:09.081861 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:37:09.074+0000 7ff641419340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:37:09.083 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:37:09.074+0000 7ff641419340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:37:09.083 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:37:09.074+0000 7ff641419340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:37:09.083 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:37:09.083 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:09.081884 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:37:09.074+0000 7ff641419340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:37:09.084 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:37:09.074+0000 7ff641419340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:37:09.084 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:37:09.074+0000 7ff641419340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:37:09.084 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:37:16.502 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:16.501687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:16.513 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:16.513795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:26.729 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:26.729676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:26.740 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:26.740704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:36.960 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:36.959424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:36.972 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:36.972179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:47.193 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:47.192744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:47.205 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:47.205004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:57.470 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:57.469873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:37:57.495 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:37:57.495557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:07.712 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:07.712399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:07.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:07.724695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:17.948 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:17.947703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:18.018 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:18.017824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:28.245 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:28.245383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:28.257 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:28.257816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:38.501 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:38.500640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:38.513 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:38.513228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:48.757 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:48.757084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:48.766 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:48.765869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:58.987 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:58.987082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:38:58.998 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:38:58.997751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:09.218 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:09.217920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:09.229 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:09.229444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:19.449 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:19.448373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:19.461 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:19.461241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:29.709 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:29.708635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:29.722 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:29.722326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:39.990 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:39.990233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:40.001 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:40.000780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:50.223 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:50.223033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:39:50.235 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:39:50.235431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:00.483 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:00.482989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:00.495 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:00.495299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:10.717 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:10.716701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:10.732 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:10.732478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:20.976 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:20.975539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:20.987 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:20.987368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:31.201 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:31.201550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:31.249 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:31.249238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:41.493 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:41.493080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:41.505 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:41.505614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:51.724 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:51.723561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:40:51.810 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:40:51.809849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:02.031 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:02.030734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:02.040 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:02.040794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:02.459 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:02.458618 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:41:02.451184161 +0000 UTC m=+13628.440549267 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:41:12.259 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:12.259090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:12.271 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:12.271398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:22.498 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:22.498438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:22.511 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:22.511090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:32.771 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:32.770800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:32.783 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:32.782758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:43.003 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:43.002683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:43.017 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:43.016777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:53.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:53.236990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:41:53.246 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:41:53.246319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:03.317 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:03.317204 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-09 02:42:03.284218363 +0000 UTC m=+13689.273583516 LastTransitionTime:2022-03-08 22:55:32 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-09T02:42:03.465 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:03.464836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:03.473 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:03.473541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:13.694 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:13.693508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:13.751 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:13.750557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:23.971 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:23.971016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:23.982 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:23.982667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:34.200 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:34.199379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:34.210 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:34.210673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:44.431 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:44.430592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:44.444 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:44.443536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:54.666 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:54.665620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:42:54.677 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:42:54.676993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:04.901 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:04.900704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:04.915 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:04.913567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:15.141 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:15.140913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:15.152 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:15.151989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:25.372 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:25.372088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:25.384 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:25.383906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:35.600 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:35.599378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:35.613 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:35.612847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:45.833 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:45.832478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:45.845 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:45.844772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:56.059 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:56.059056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:43:56.073 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:43:56.072537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:06.293 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:06.292807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:06.305 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:06.304979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:16.524 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:16.523186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:16.535 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:16.534990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:26.755 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:26.754532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:26.772 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:26.770039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:37.018 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:37.017433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:37.032 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:37.030941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:47.248 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:47.248334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:47.263 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:47.262662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:57.482 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:57.481833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:44:57.498 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:44:57.497754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:07.719 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:07.719156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:07.727 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:07.727248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:17.941 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:17.940916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:18.036 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:18.036096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:28.284 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:28.283475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:28.296 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:28.295991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:38.518 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:38.517644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:38.530 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:38.530421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:48.751 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:48.751227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:48.763 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:48.763645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:58.987 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:58.986785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:45:59.000 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:45:59.000198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:09.222 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:09.221807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:09.237 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:09.237059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:19.462 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:19.462158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:19.474 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:19.473422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:29.701 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:29.701016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:29.709 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:29.708421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:39.938 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:39.937803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:39.950 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:39.950515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:50.171 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:50.170992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:46:50.185 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:46:50.184634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:00.406 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:00.405510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:00.419 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:00.418984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:10.643 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:10.642753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:10.656 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:10.656201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:20.877 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:20.876714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:20.884 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:20.884855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:31.110 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:31.109554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:31.122 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:31.122255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:41.370 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:41.369471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:41.381 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:41.381183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:51.599 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:51.598997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:47:51.612 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:47:51.611880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:01.835 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:01.834413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:01.852 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:01.845238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:12.065 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:12.064673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:12.072 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:12.072490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:22.291 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:22.290675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:22.330 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:22.329716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:32.567 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:32.566752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:32.575 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:32.575300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:42.809 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:42.808128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:42.820 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:42.820150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:53.036 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:53.036087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:48:53.059 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:48:53.059313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:03.279 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:03.279109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:03.294 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:03.293593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:13.605 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:13.604883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:13.615 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:13.614593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:23.827 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:23.826840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:23.838 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:23.838413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:34.055 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:34.054462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:34.065 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:34.065366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:44.286 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:44.285749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:44.296 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:44.296527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:54.557 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:54.556868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:49:54.568 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:49:54.568276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:04.788 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:04.787548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:04.801 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:04.800484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:15.023 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:15.022071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:15.036 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:15.036195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:25.259 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:25.258777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:25.271 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:25.270536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:35.492 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:35.492085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:35.503 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:35.503079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:45.725 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:45.724227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:45.739 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:45.737906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:55.960 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:55.959113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:50:55.975 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:50:55.973957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:06.195 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:06.194631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:06.209 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:06.206523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:16.426 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:16.424881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:16.437 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:16.436608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:26.692 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:26.690806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:26.707 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:26.704529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:36.925 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:36.924938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:36.935 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:36.934617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:47.155 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:47.154803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:47.168 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:47.168171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:57.394 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:57.394136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:51:57.409 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:51:57.408562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:07.629 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:07.628094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:07.642 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:07.641534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:17.865 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:17.864683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:17.878 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:17.878064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:28.114 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:28.113494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:28.125 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:28.124323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:38.384 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:38.383974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:38.397 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:38.396587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:48.616 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:48.615586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:48.631 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:48.623766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:58.844 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:58.843097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:52:58.857 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:52:58.856882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:09.078 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:09.077607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:09.091 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:09.090201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:19.331 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:19.330374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:19.340 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:19.339657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:29.582 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:29.581967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:29.594 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:29.594459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:39.813 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:39.812221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:39.826 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:39.826179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:49.091 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:49.090155 I | op-mon: parsing mon endpoints: a=172.21.15.176:6789 2022-03-09T02:53:49.697 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:49.695996 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:53:49.684+0000 7f7e7b397340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:53:49.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:53:49.684+0000 7f7e7b397340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:53:49.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:53:49.684+0000 7f7e7b397340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:53:49.698 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:53:49.698 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:49.696048 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:53:49.684+0000 7f7e7b397340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:53:49.699 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:53:49.684+0000 7f7e7b397340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:53:49.699 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09T02:53:49.684+0000 7f7e7b397340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:53:49.699 INFO:tasks.rook.operator.smithi176.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:53:50.043 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:50.042809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:53:50.055 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:53:50.054772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:00.277 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:54:00.277009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T02:54:00.290 INFO:tasks.rook.operator.smithi176.stdout:2022-03-09 02:54:00.288652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-09T05:15:00.565 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-09T05:15:00.608 DEBUG:teuthology.task.console_log:Killing console logger for smithi176 2022-03-09T05:15:00.609 DEBUG:teuthology.exit:Finished running handlers